Skip to main content

Register ArcGIS Enterprise on Kubernetes

You can register ArcGIS Enterprise on Kubernetes to collect system health, performance, and application usage metrics.

ArcGIS Monitor uses remote monitoring for ArcGIS Enterprise on Kubernetes. Install ArcGIS Monitor Agent on a machine outside the Kubernetes cluster that can access the ArcGIS Enterprise on Kubernetes URL and, when configured, the Prometheus and Kubernetes API endpoints.

Prerequisites for monitoring

The following prerequisites are required to monitor ArcGIS Enterprise on Kubernetes:

  • ArcGIS Monitor Agent must be installed on a machine with network access to ArcGIS Enterprise on Kubernetes and must be registered with ArcGIS Monitor Server.

  • ArcGIS Enterprise on Kubernetes 12.2 is supported for monitoring. This version is the minimum certified version. Newer minor versions are supported, but are not certified unless otherwise stated.

  • To register ArcGIS Enterprise on Kubernetes, use an organization administrator account. Monitor uses this account to access the ArcGIS Enterprise administrative and sharing APIs.

  • Authentication requires ArcGIS token-based security. Other forms of authentication, such as Integrated Windows Authentication (IWA) and Security Assertion Markup Language (SAML), are not supported.

  • To collect service request and timing metrics, obtain the username and password for the Prometheus endpoint. Monitor derives the endpoint URL from the ArcGIS Enterprise on Kubernetes URL; you do not need to provide a Prometheus URL.

  • To collect Kubernetes infrastructure and resource utilization metrics, you can configure Kubernetes API access. Provide the Kubernetes API server URL and the service account bearer token when you register the component. Monitor discovers the ArcGIS Enterprise on Kubernetes namespace; you do not need to provide a namespace value.

Configure Kubernetes API access

Kubernetes API access is optional. When configured, Monitor collects pod phase and count metrics, CPU and memory utilization metrics, ResourceQuota limit utilization metrics, and cluster node utilization metrics.

To configure Kubernetes API access, you must complete the following processes:

  • Verify that the Kubernetes Metrics API is available

  • Create a service account and its role-based access control (RBAC) resources

  • Generate a token for the service account

  • Determine the Kubernetes API endpoint

  • Validate the assigned permissions

To complete these processes, you can run the commands in the following sections with the Kubernetes command-line tool (kubectl) from a machine that is configured to access your cluster. The account you use must have permission to create RBAC resources and secrets in the cluster.

Verify that the Kubernetes Metrics API is available

Monitor collects CPU and memory utilization from the metrics.k8s.io API, which requires Metrics Server to be installed in the cluster. To verify that the Metrics API is available, run the following commands, where <ArcGIS Enterprise on Kubernetes namespace> is the namespace that contains your ArcGIS Enterprise on Kubernetes deployment:

kubectl top nodes
kubectl top pods -n <ArcGIS Enterprise on Kubernetes namespace>

If either command returns an error similar to the following, install or troubleshoot Metrics Server before you continue:

error: Metrics API not available

Create the service account and RBAC resources

The following example creates a dedicated service account with the baseline permissions required for Kubernetes metrics. The manifest uses the following placeholders:

  • <ArcGIS Enterprise on Kubernetes namespace>—The namespace that contains your ArcGIS Enterprise on Kubernetes deployment.

  • <service account name>—The name of the service account that Monitor uses, for example, arcgis-monitor. Use a value that follows your organization's naming conventions.

  • <role name>—The name of the namespace-scoped Role and RoleBinding, for example, arcgis-monitor-metrics.

  • <cluster role name>—The name of the ClusterRole and ClusterRoleBinding, for example, arcgis-monitor-metrics.

Save the following manifest as arcgis-monitor-rbac.yaml, replacing the placeholders with the specified details:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: <service account name>
  namespace: <ArcGIS Enterprise on Kubernetes namespace>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <role name>
  namespace: <ArcGIS Enterprise on Kubernetes namespace>
rules:
  - apiGroups: [""]
    resources: ["pods", "resourcequotas"]
    verbs: ["get", "list"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <role name>
  namespace: <ArcGIS Enterprise on Kubernetes namespace>
subjects:
  - kind: ServiceAccount
    name: <service account name>
    namespace: <ArcGIS Enterprise on Kubernetes namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: <role name>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: <cluster role name>
rules:
  - apiGroups: [""]
    resources: ["nodes", "namespaces"]
    verbs: ["get", "list"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["nodes"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: <cluster role name>
subjects:
  - kind: ServiceAccount
    name: <service account name>
    namespace: <ArcGIS Enterprise on Kubernetes namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: <cluster role name>
Important:

The apiVersion, kind, apiGroups, resources, and verbs values are the minimum permissions that Monitor requires. Do not change these values. Access to namespaces is required because Monitor retrieves namespace information as part of metrics collection. Monitor uses this access to discover the namespace that contains your ArcGIS Enterprise on Kubernetes deployment.

To apply the manifest, run the following command:

kubectl apply -f arcgis-monitor-rbac.yaml

The ClusterRole grants access to cluster-scoped node and namespace data. All other baseline permissions are limited to the ArcGIS Enterprise on Kubernetes namespace.

If you monitor more than one ArcGIS Enterprise on Kubernetes deployment in the same cluster, create a service account, Role, and RoleBinding in each namespace. You can reuse one ClusterRole, but each service account requires its own entry in the subjects list of the ClusterRoleBinding.

Generate a service account token

Monitor authenticates to the Kubernetes API with a service account bearer token. To generate a token, run the following command:

kubectl -n <ArcGIS Enterprise on Kubernetes namespace> create token <service account name>
Note:

Tokens created with kubectl create token expire one hour after they are issued by default. When the token expires, the Metrics - ArcGIS Enterprise on Kubernetes observer can no longer collect Kubernetes metrics until you provide a new token. Use this command for testing, or specify a longer lifetime with the --duration parameter.

For ongoing monitoring, create a long-lived token instead. Kubernetes 1.24 and later versions do not automatically create a secret for a service account, so you must create one explicitly. The manifest uses the following placeholders:

  • <token secret name>—The name of the secret that stores the token, for example, arcgis-monitor-token.

  • <ArcGIS Enterprise on Kubernetes namespace>—The namespace that contains your ArcGIS Enterprise on Kubernetes deployment. Use the same value that you used in the RBAC manifest.

  • <service account name>—The name of the service account that Monitor uses, for example, arcgis-monitor. Use the same value that you used in the RBAC manifest.

Save the following manifest as arcgis-monitor-token.yaml, replacing the placeholders with the specified details:

apiVersion: v1
kind: Secret
metadata:
  name: <token secret name>
  namespace: <ArcGIS Enterprise on Kubernetes namespace>
  annotations:
    kubernetes.io/service-account.name: <service account name>
type: kubernetes.io/service-account-token
Note:

The type value and the kubernetes.io/service-account.name annotation key are required for Kubernetes to populate the secret with a token. Do not change them.

To apply the manifest, run the following command:

kubectl apply -f arcgis-monitor-token.yaml

To retrieve the decoded token value, run the following command:

kubectl -n <ArcGIS Enterprise on Kubernetes namespace> get secret <token secret name> -o go-template="{{.data.token | base64decode}}"
Caution:

A token issued from a service account secret does not expire. Store the token securely, restrict access to the secret, and delete the secret to revoke access.

Determine the Kubernetes API endpoint

To retrieve the Kubernetes API server URL, run the following command:

kubectl config view --minify -o jsonpath="{.clusters[0].cluster.server}"

The command returns the API server URL. The following is an example of a returned URL:

https://my-cluster.example.com:6443

Use this value as the Kubernetes API server URL when you register the component.

Validate service account permissions

To verify that the service account has the required permissions, run the following commands:

kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list pods -n <ArcGIS Enterprise on Kubernetes namespace>
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list resourcequotas -n <ArcGIS Enterprise on Kubernetes namespace>
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> get namespaces
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list nodes
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list pods.metrics.k8s.io -n <ArcGIS Enterprise on Kubernetes namespace>
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list nodes.metrics.k8s.io

Each command returns yes when the permission is granted.

Configure application usage sessions

The Sessions - ArcGIS Enterprise on Kubernetes observer collects application usage from ingress controller logs. To enable sessions collection, add the following rules to the rules list of the namespace-scoped role created for the service account and RBAC resources:

- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["arcgis-ingress-controller-nginx-config"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create", "get"]

The resourceNames value is defined by ArcGIS Enterprise on Kubernetes; do not change it.

Caution:

The pods/exec permission allows the service account to run commands in any pod in the ArcGIS Enterprise on Kubernetes namespace. Add this permission only if you want to collect application usage sessions. Monitor uses it only to run commands in ingress controller pods labeled with arcgis/app=ingress-nginx.

To verify that the service account has the sessions permissions, run the following commands:

kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> update configmaps/arcgis-ingress-controller-nginx-config -n <ArcGIS Enterprise on Kubernetes namespace>
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> create pods/exec -n <ArcGIS Enterprise on Kubernetes namespace>

Each command returns yes when the permission is granted.

When enabled, the observer adds Monitor-managed logging entries to the ingress controller ConfigMap, reads and rotates a dedicated log in ingress controller pods, and removes only its own logging entries when disabled.

Register ArcGIS Enterprise on Kubernetes for monitoring

You must be assigned the Administrator or Manager role to register components.

To register an ArcGIS Enterprise on Kubernetes component, complete the following steps:

  1. Access ArcGIS Monitor if necessary.

    The Home page appears.

  2. Click Monitoring.

    The Monitoring page appears with registered components listed alphabetically by name.

  3. Click Register component.

    The Register component wizard appears.

    Note:

    If you don't have sufficient privileges, Register component is not available.

  4. Choose the Monitor Agent machine that can access ArcGIS Enterprise on Kubernetes, and click Next.

    The Component type page appears.

  5. Under ArcGIS, choose ArcGIS Enterprise on Kubernetes, and click Next.

    The Component details page appears.

  6. Provide a name for the component in the Component name text box.
  7. Optionally, provide a description for the component in the Description text box.
  8. Optionally, click Apply Labels and choose the labels that you want to apply to the component.

    Tip:

    Provide the name of a label in the Enter label name text box to search for a specific label. If the label doesn't exist, click Create new label to add the label and apply it to the component.

  9. Click Next.

    The Connection page appears.

  10. Under ArcGIS credentials, provide the ArcGIS Enterprise on Kubernetes URL and the username and password for an organization administrator.

  11. Optionally, under Prometheus credentials, provide the Prometheus username and password to collect service request, response time, wait time, error, and instance metrics.

  12. Optionally, under Kubernetes credentials, provide the Kubernetes API server URL and the bearer token for the service account that you configured.

  13. Click Register component.

    The component is added to the list of monitored components on the Monitoring page.

Once a component is registered, you can configure alert rules and add the component to collections.

Observers

The following table describes the system-managed observers that collect ArcGIS Enterprise on Kubernetes data:

Observer name

Description

Inventory - ArcGIS Enterprise on Kubernetes

This observer scans ArcGIS Enterprise on Kubernetes for changes and collects attributes for the deployment, GIS services, web maps, and web applications. It also manages relationships between these components.

The default interval for this observer is 5 minutes.

Logs - ArcGIS Enterprise on Kubernetes

This observer collects warning and severe administrative log entries from ArcGIS Enterprise on Kubernetes.

The default interval for this observer is 1 minute. Do not run this observer at longer intervals.

Metrics - ArcGIS Enterprise on Kubernetes

This observer collects request, availability, pod, and resource utilization metrics. The data available depends on the connection information you provide.

The default interval for this observer is 1 minute.

Sessions - ArcGIS Enterprise on Kubernetes

This observer collects application usage sessions for supported web applications and web maps. Kubernetes API credentials with the additional sessions permissions are required.

The default interval for this observer is 1 minute. Do not run this observer at longer intervals.

Collected attributes

ArcGIS Enterprise on Kubernetes

The following table lists the attributes that are collected for ArcGIS Enterprise on Kubernetes:

Attribute name

Description

Data type

Availability profile

The availability architecture profile configured for the deployment

String

Performance profile

The performance architecture profile configured for the deployment

String

Certificate expiration

The date and time the configured ingress certificate expires

Datetime

License expiration

The date and time the earliest ArcGIS Enterprise on Kubernetes license expires

Datetime

Roles

The list of ArcGIS Server roles available in the deployment

Array

Started at

The date and time ArcGIS Enterprise on Kubernetes was last started

Datetime

Version

The software version of ArcGIS Enterprise on Kubernetes

String

ArcGIS Enterprise on Kubernetes services

The following table lists the attributes that are collected for ArcGIS Enterprise on Kubernetes services:

Attribute name

Description

Data type

Cached service

Specifies whether data is returned dynamically or returned from a cache when queried

Boolean

Configured state

Specifies the state of the service as it is configured in ArcGIS Enterprise on Kubernetes (started or stopped)

String

Instance type

Specifies the infrastructure of the underlying service provider (hosted, shared, or dedicated)

String

Idle time max

The maximum number of seconds a service instance is allowed to persist before it shuts down

Integer

Is remote

Specifies whether the service is remote

Boolean

Pods max

The maximum number of pods configured for the service

Integer

Pods min

The minimum number of pods configured for the service

Integer

Pods scaling mode

The configured pod scaling mode

String

Wait time max

The maximum number of seconds a client waits to connect to the service before it times out

Integer

Web applications

The following table lists the attributes that are collected for web applications:

Attribute name

Description

Data type

Content status

The content status of the web application

String

Last modified at

The date and time the web application was last modified

Datetime

Owner

The owner of the web application

String

Sharing level

The shared access level of the web application

String

System created at

The date and time the web application was created

Datetime

Web maps

The following table lists the attributes that are collected for web maps:

Attribute name

Description

Data type

Content status

The content status of the web map

String

Last modified at

The date and time the web map was last modified

Datetime

Owner

The owner of the web map

String

Sharing level

The shared access level of the web map

String

System created at

The date and time the web map was created

Datetime

Collected metrics

ArcGIS Enterprise on Kubernetes

The following table identifies the metric types collected for the ArcGIS Enterprise on Kubernetes namespace and its related infrastructure.

Entity

Resource usage

Pod phases

Instance usage

Requests

Namespace

Shared service pools

Infrastructure

Cluster

The following table lists the metrics that are collected for ArcGIS Enterprise on Kubernetes:

Metric name

Description

Data type

Unit

Active Sessions

The total number of active sessions over the interval for all web applications and web maps in ArcGIS Enterprise on Kubernetes.

Float

Total

Active Sessions Time

The duration of all active sessions in all web applications and web maps in ArcGIS Enterprise on Kubernetes over the interval.

Float

Time

Connectivity

The agent's ability to successfully connect to the component.

Float

Percent

Instances Saturation Percent

The percentage of maximum available service instances in use.

This metric is calculated by dividing the service's busy time with its maximum potential busy time, and multiplying this value by 100.

The service busy time is calculated by multiplying the service's average response time (RT) by the number of requests (RR) over the interval.

The maximum potential busy time is calculated by multiplying the currently available service instances (IC) by the interval (I). ArcGIS Enterprise on Kubernetes derives the available service instances from the number of running pods.

The following is an example:

(RT*RR)/(IC*I)*100
(0.1*1000)/(2*60)*100
83% of maximum available service instances

Values approaching 100 percent may indicate that more instances are needed to handle incoming service requests.

Float

Percent

Instances Used

The total number of service instances in use.

This metric is calculated by dividing the instances saturation percent metric value (ISP) by 100, and multiplying this value by the currently available service instances (IC) as derived by the number of running pods.

The following is an example:

(ISP/100)*IC
(83/100)*2
1.66 instances used

Float

Total

Request Rate

The average request rate of all ArcGIS Enterprise on Kubernetes services.

Float

Requests/second

Request Response Time Avg

The average response time of all ArcGIS Enterprise on Kubernetes services.

Float

Seconds

Request Response Time Max

The maximum response time of all ArcGIS Enterprise on Kubernetes services.

Float

Seconds

Request Wait Time Avg

The average time that all requests waited for an available service instance over the specified period.

Float

Seconds

Request Wait Time Max

The maximum time that a service request waited for an available service instance over the specified period.

Float

Seconds

Request Wait Time Min

The minimum time that a service request waited for an available service instance over the specified period.

Float

Seconds

Requests Error Percent

The percentage of failed requests of all ArcGIS Enterprise on Kubernetes services.

Float

Percent

Requests Failed

The total number of failed requests of all ArcGIS Enterprise on Kubernetes services.

Float

Total

Requests Received

The total number of received requests of all ArcGIS Enterprise on Kubernetes services.

Float

Total

Services

The total number of ArcGIS Enterprise on Kubernetes services.

Float

Total

Pods

The total number of pods.

Float

Pods

Pods Running

The total number of pods in the Running phase.

Float

Pods

Pods Pending

The total number of pods in the Pending phase.

Float

Pods

Pods Failed

The total number of pods in the Failed phase.

Float

Pods

Pods Unknown

The total number of pods in the Unknown phase.

Float

Pods

ArcGIS Enterprise on Kubernetes services metrics

Monitor collects information about the following ArcGIS Server service types:

  • Feature

  • Geocode

  • Geodata

  • Geoprocessing

  • Image

  • Map

  • Network Analyst

All metrics in this table are collected for services with a dedicated instance type. For services with a shared or hosted instance type, Monitor collects only request and instance usage metrics; their pod and resource usage metrics are collected for the related shared service pool.

The following table lists the metrics that are collected for ArcGIS Enterprise on Kubernetes services:

Metric name

Description

Data type

Unit

Instances Saturation Percent

The percentage of maximum configured service instances in use as derived by the number of running pods.

Values approaching 100 percent may indicate that more instances are needed to handle incoming service requests.

Float

Percent

Instances Used

The total number of service instances in use.

Float

Total

Request Rate

The service's average request rate.

Float

Requests/second

Request Response Time Avg

The service's average response time.

Float

Seconds

Request Response Time Max

The service's maximum response time.

Float

Seconds

Request Wait Time Avg

The average time that service requests waited for an available service instance over the specified period.

Float

Seconds

Request Wait Time Max

The maximum time that a service request waited for an available service instance over the specified period.

Float

Seconds

Request Wait Time Min

The minimum time that a service request waited for an available service instance over the specified period.

Float

Seconds

Requests Error Percent

The percentage of failed service requests.

Float

Percent

Requests Failed

The total number of failed service requests.

Float

Total

Requests Received

The total number of received service requests.

Float

Total

Pods

The total number of pods for the service.

Float

Pods

Pods Running

The total number of pods for the service in the Running phase.

Float

Pods

Pods Pending

The total number of pods for the service in the Pending phase.

Float

Pods

Pods Failed

The total number of pods for the service in the Failed phase.

Float

Pods

Pods Unknown

The total number of pods for the service in the Unknown phase.

Float

Pods

CPU Utilized

The proportion of schedulable cluster CPU capacity in use by the service.

Float

Percent

CPU Limit Utilized

The proportion of the service's configured CPU limit in use.

Float

Percent

Memory Utilized

The proportion of schedulable cluster memory capacity in use by the service.

Float

Percent

Memory Limit Utilized

The proportion of the service's configured memory limit in use.

Float

Percent

Web applications and web maps

The following table lists the metrics that are collected for web applications and web maps:

Metric name

Description

Data type

Unit

Active Sessions

The total number of active sessions over the interval for the web application or web map

Float

Total

Active Sessions Time

The duration of all active sessions for the web application or web map over the interval

Float

Time

Note:

Kubernetes API access is optional. Without it, Kubernetes-derived pod and resource utilization metrics and application usage sessions are not collected. Prometheus credentials are also optional; without them, service request metrics are not collected.

Component relationships

The inventory observer creates relationships between the ArcGIS Enterprise on Kubernetes component and its GIS services, web maps, and web applications. It also creates relationships from web maps and web applications to the services and web maps they use when the related items can be resolved.