Register ArcGIS Enterprise on Kubernetes
You can register ArcGIS Enterprise on Kubernetes to collect system health, performance, and application usage metrics.
ArcGIS Monitor uses remote monitoring for ArcGIS Enterprise on Kubernetes. Install ArcGIS Monitor Agent on a machine outside the Kubernetes cluster that can access the ArcGIS Enterprise on Kubernetes URL and, when configured, the Prometheus and Kubernetes API endpoints.
Prerequisites for monitoring
The following prerequisites are required to monitor ArcGIS Enterprise on Kubernetes:
ArcGIS Monitor Agent must be installed on a machine with network access to ArcGIS Enterprise on Kubernetes and must be registered with ArcGIS Monitor Server.
ArcGIS Enterprise on Kubernetes 12.2 is supported for monitoring. This version is the minimum certified version. Newer minor versions are supported, but are not certified unless otherwise stated.
To register ArcGIS Enterprise on Kubernetes, use an organization administrator account. Monitor uses this account to access the ArcGIS Enterprise administrative and sharing APIs.
Authentication requires ArcGIS token-based security. Other forms of authentication, such as Integrated Windows Authentication (IWA) and Security Assertion Markup Language (SAML), are not supported.
To collect service request and timing metrics, obtain the username and password for the Prometheus endpoint. Monitor derives the endpoint URL from the ArcGIS Enterprise on Kubernetes URL; you do not need to provide a Prometheus URL.
To collect Kubernetes infrastructure and resource utilization metrics, you can configure Kubernetes API access. Provide the Kubernetes API server URL and the service account bearer token when you register the component. Monitor discovers the ArcGIS Enterprise on Kubernetes namespace; you do not need to provide a namespace value.
Configure Kubernetes API access
Kubernetes API access is optional. When configured, Monitor collects pod phase and count metrics, CPU and memory utilization metrics, ResourceQuota limit utilization metrics, and cluster node utilization metrics.
To configure Kubernetes API access, you must complete the following processes:
Verify that the Kubernetes Metrics API is available
Create a service account and its role-based access control (RBAC) resources
Generate a token for the service account
Determine the Kubernetes API endpoint
Validate the assigned permissions
To complete these processes, you can run the commands in the following sections with the Kubernetes command-line tool (kubectl) from a machine that is configured to access your cluster. The account you use must have permission to create RBAC resources and secrets in the cluster.
Verify that the Kubernetes Metrics API is available
Monitor collects CPU and memory utilization from the metrics.k8s.io API, which requires Metrics Server to be installed in the cluster. To verify that the Metrics API is available, run the following commands, where <ArcGIS Enterprise on Kubernetes namespace> is the namespace that contains your ArcGIS Enterprise on Kubernetes deployment:
kubectl top nodes
kubectl top pods -n <ArcGIS Enterprise on Kubernetes namespace>
If either command returns an error similar to the following, install or troubleshoot Metrics Server before you continue:
error: Metrics API not available
Create the service account and RBAC resources
The following example creates a dedicated service account with the baseline permissions required for Kubernetes metrics. The manifest uses the following placeholders:
<ArcGIS Enterprise on Kubernetes namespace>—The namespace that contains your ArcGIS Enterprise on Kubernetes deployment.<service account name>—The name of the service account that Monitor uses, for example,arcgis-monitor. Use a value that follows your organization's naming conventions.<role name>—The name of the namespace-scoped Role and RoleBinding, for example,arcgis-monitor-metrics.<cluster role name>—The name of the ClusterRole and ClusterRoleBinding, for example,arcgis-monitor-metrics.
Save the following manifest as arcgis-monitor-rbac.yaml, replacing the placeholders with the specified details:
apiVersion: v1
kind: ServiceAccount
metadata:
name: <service account name>
namespace: <ArcGIS Enterprise on Kubernetes namespace>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: <role name>
namespace: <ArcGIS Enterprise on Kubernetes namespace>
rules:
- apiGroups: [""]
resources: ["pods", "resourcequotas"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: <role name>
namespace: <ArcGIS Enterprise on Kubernetes namespace>
subjects:
- kind: ServiceAccount
name: <service account name>
namespace: <ArcGIS Enterprise on Kubernetes namespace>
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: <role name>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: <cluster role name>
rules:
- apiGroups: [""]
resources: ["nodes", "namespaces"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["nodes"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: <cluster role name>
subjects:
- kind: ServiceAccount
name: <service account name>
namespace: <ArcGIS Enterprise on Kubernetes namespace>
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: <cluster role name>
Important:
The apiVersion, kind, apiGroups, resources, and verbs values are the minimum permissions that Monitor requires. Do not change these values. Access to namespaces is required because Monitor retrieves namespace information as part of metrics collection. Monitor uses this access to discover the namespace that contains your ArcGIS Enterprise on Kubernetes deployment.
To apply the manifest, run the following command:
kubectl apply -f arcgis-monitor-rbac.yaml
The ClusterRole grants access to cluster-scoped node and namespace data. All other baseline permissions are limited to the ArcGIS Enterprise on Kubernetes namespace.
If you monitor more than one ArcGIS Enterprise on Kubernetes deployment in the same cluster, create a service account, Role, and RoleBinding in each namespace. You can reuse one ClusterRole, but each service account requires its own entry in the subjects list of the ClusterRoleBinding.
Generate a service account token
Monitor authenticates to the Kubernetes API with a service account bearer token. To generate a token, run the following command:
kubectl -n <ArcGIS Enterprise on Kubernetes namespace> create token <service account name>
Note:
Tokens created with kubectl create token expire one hour after they are issued by default. When the token expires, the Metrics - ArcGIS Enterprise on Kubernetes observer can no longer collect Kubernetes metrics until you provide a new token. Use this command for testing, or specify a longer lifetime with the --duration parameter.
For ongoing monitoring, create a long-lived token instead. Kubernetes 1.24 and later versions do not automatically create a secret for a service account, so you must create one explicitly. The manifest uses the following placeholders:
<token secret name>—The name of the secret that stores the token, for example,arcgis-monitor-token.<ArcGIS Enterprise on Kubernetes namespace>—The namespace that contains your ArcGIS Enterprise on Kubernetes deployment. Use the same value that you used in the RBAC manifest.<service account name>—The name of the service account that Monitor uses, for example,arcgis-monitor. Use the same value that you used in the RBAC manifest.
Save the following manifest as arcgis-monitor-token.yaml, replacing the placeholders with the specified details:
apiVersion: v1
kind: Secret
metadata:
name: <token secret name>
namespace: <ArcGIS Enterprise on Kubernetes namespace>
annotations:
kubernetes.io/service-account.name: <service account name>
type: kubernetes.io/service-account-token
Note:
The type value and the kubernetes.io/service-account.name annotation key are required for Kubernetes to populate the secret with a token. Do not change them.
To apply the manifest, run the following command:
kubectl apply -f arcgis-monitor-token.yaml
To retrieve the decoded token value, run the following command:
kubectl -n <ArcGIS Enterprise on Kubernetes namespace> get secret <token secret name> -o go-template="{{.data.token | base64decode}}"
Caution:
A token issued from a service account secret does not expire. Store the token securely, restrict access to the secret, and delete the secret to revoke access.
Determine the Kubernetes API endpoint
To retrieve the Kubernetes API server URL, run the following command:
kubectl config view --minify -o jsonpath="{.clusters[0].cluster.server}"
The command returns the API server URL. The following is an example of a returned URL:
https://my-cluster.example.com:6443
Use this value as the Kubernetes API server URL when you register the component.
Validate service account permissions
To verify that the service account has the required permissions, run the following commands:
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list pods -n <ArcGIS Enterprise on Kubernetes namespace>
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list resourcequotas -n <ArcGIS Enterprise on Kubernetes namespace>
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> get namespaces
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list nodes
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list pods.metrics.k8s.io -n <ArcGIS Enterprise on Kubernetes namespace>
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> list nodes.metrics.k8s.io
Each command returns yes when the permission is granted.
Configure application usage sessions
The Sessions - ArcGIS Enterprise on Kubernetes observer collects application usage from ingress controller logs. To enable sessions collection, add the following rules to the rules list of the namespace-scoped role created for the service account and RBAC resources:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["arcgis-ingress-controller-nginx-config"]
verbs: ["get", "update"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create", "get"]
The resourceNames value is defined by ArcGIS Enterprise on Kubernetes; do not change it.
Caution:
The pods/exec permission allows the service account to run commands in any pod in the ArcGIS Enterprise on Kubernetes namespace. Add this permission only if you want to collect application usage sessions. Monitor uses it only to run commands in ingress controller pods labeled with arcgis/app=ingress-nginx.
To verify that the service account has the sessions permissions, run the following commands:
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> update configmaps/arcgis-ingress-controller-nginx-config -n <ArcGIS Enterprise on Kubernetes namespace>
kubectl auth can-i --as=system:serviceaccount:<ArcGIS Enterprise on Kubernetes namespace>:<service account name> create pods/exec -n <ArcGIS Enterprise on Kubernetes namespace>
Each command returns yes when the permission is granted.
When enabled, the observer adds Monitor-managed logging entries to the ingress controller ConfigMap, reads and rotates a dedicated log in ingress controller pods, and removes only its own logging entries when disabled.
Register ArcGIS Enterprise on Kubernetes for monitoring
You must be assigned the Administrator or Manager role to register components.
To register an ArcGIS Enterprise on Kubernetes component, complete the following steps:
Access ArcGIS Monitor if necessary.
The Home page appears.
Click Monitoring.
The Monitoring page appears with registered components listed alphabetically by name.
Click Register component.
The Register component wizard appears.
Note:
If you don't have sufficient privileges, Register component is not available.
Choose the Monitor Agent machine that can access ArcGIS Enterprise on Kubernetes, and click Next.
The Component type page appears.
Under ArcGIS, choose ArcGIS Enterprise on Kubernetes, and click Next.
The Component details page appears.
- Provide a name for the component in the Component name text box.
- Optionally, provide a description for the component in the Description text box.
Optionally, click Apply Labels
and choose the labels that you want to apply to the component.Tip:
Provide the name of a label in the Enter label name text box to search for a specific label. If the label doesn't exist, click Create new label to add the label and apply it to the component.
Click Next.
The Connection page appears.
Under ArcGIS credentials, provide the ArcGIS Enterprise on Kubernetes URL and the username and password for an organization administrator.
Optionally, under Prometheus credentials, provide the Prometheus username and password to collect service request, response time, wait time, error, and instance metrics.
Optionally, under Kubernetes credentials, provide the Kubernetes API server URL and the bearer token for the service account that you configured.
Click Register component.
The component is added to the list of monitored components on the Monitoring page.
Once a component is registered, you can configure alert rules and add the component to collections.
Observers
The following table describes the system-managed observers that collect ArcGIS Enterprise on Kubernetes data:
|
Observer name |
Description |
|---|---|
|
Inventory - ArcGIS Enterprise on Kubernetes |
This observer scans ArcGIS Enterprise on Kubernetes for changes and collects attributes for the deployment, GIS services, web maps, and web applications. It also manages relationships between these components. The default interval for this observer is 5 minutes. |
|
Logs - ArcGIS Enterprise on Kubernetes |
This observer collects warning and severe administrative log entries from ArcGIS Enterprise on Kubernetes. The default interval for this observer is 1 minute. Do not run this observer at longer intervals. |
|
Metrics - ArcGIS Enterprise on Kubernetes |
This observer collects request, availability, pod, and resource utilization metrics. The data available depends on the connection information you provide. The default interval for this observer is 1 minute. |
|
Sessions - ArcGIS Enterprise on Kubernetes |
This observer collects application usage sessions for supported web applications and web maps. Kubernetes API credentials with the additional sessions permissions are required. The default interval for this observer is 1 minute. Do not run this observer at longer intervals. |
Collected attributes
ArcGIS Enterprise on Kubernetes
The following table lists the attributes that are collected for ArcGIS Enterprise on Kubernetes:
|
Attribute name |
Description |
Data type |
|---|---|---|
|
Availability profile |
The availability architecture profile configured for the deployment |
String |
|
Performance profile |
The performance architecture profile configured for the deployment |
String |
|
Certificate expiration |
The date and time the configured ingress certificate expires |
Datetime |
|
License expiration |
The date and time the earliest ArcGIS Enterprise on Kubernetes license expires |
Datetime |
|
Roles |
The list of ArcGIS Server roles available in the deployment |
Array |
|
Started at |
The date and time ArcGIS Enterprise on Kubernetes was last started |
Datetime |
|
Version |
The software version of ArcGIS Enterprise on Kubernetes |
String |
ArcGIS Enterprise on Kubernetes services
The following table lists the attributes that are collected for ArcGIS Enterprise on Kubernetes services:
|
Attribute name |
Description |
Data type |
|---|---|---|
|
Cached service |
Specifies whether data is returned dynamically or returned from a cache when queried |
Boolean |
|
Configured state |
Specifies the state of the service as it is configured in ArcGIS Enterprise on Kubernetes (started or stopped) |
String |
|
Instance type |
Specifies the infrastructure of the underlying service provider (hosted, shared, or dedicated) |
String |
|
Idle time max |
The maximum number of seconds a service instance is allowed to persist before it shuts down |
Integer |
|
Is remote |
Specifies whether the service is remote |
Boolean |
|
Pods max |
The maximum number of pods configured for the service |
Integer |
|
Pods min |
The minimum number of pods configured for the service |
Integer |
|
Pods scaling mode |
The configured pod scaling mode |
String |
|
Wait time max |
The maximum number of seconds a client waits to connect to the service before it times out |
Integer |
Web applications
The following table lists the attributes that are collected for web applications:
|
Attribute name |
Description |
Data type |
|---|---|---|
|
Content status |
The content status of the web application |
String |
|
Last modified at |
The date and time the web application was last modified |
Datetime |
|
Owner |
The owner of the web application |
String |
|
Sharing level |
The shared access level of the web application |
String |
|
System created at |
The date and time the web application was created |
Datetime |
Web maps
The following table lists the attributes that are collected for web maps:
|
Attribute name |
Description |
Data type |
|---|---|---|
|
Content status |
The content status of the web map |
String |
|
Last modified at |
The date and time the web map was last modified |
Datetime |
|
Owner |
The owner of the web map |
String |
|
Sharing level |
The shared access level of the web map |
String |
|
System created at |
The date and time the web map was created |
Datetime |
Collected metrics
ArcGIS Enterprise on Kubernetes
The following table identifies the metric types collected for the ArcGIS Enterprise on Kubernetes namespace and its related infrastructure.
|
Entity |
Resource usage |
Pod phases |
Instance usage |
Requests |
|---|---|---|---|---|
|
Namespace |
|
|
|
|
|
Shared service pools |
|
|
| |
|
Infrastructure |
|
| ||
|
Cluster |
|
The following table lists the metrics that are collected for ArcGIS Enterprise on Kubernetes:
|
Metric name |
Description |
Data type |
Unit |
|---|---|---|---|
|
Active Sessions |
The total number of active sessions over the interval for all web applications and web maps in ArcGIS Enterprise on Kubernetes. |
Float |
Total |
|
Active Sessions Time |
The duration of all active sessions in all web applications and web maps in ArcGIS Enterprise on Kubernetes over the interval. |
Float |
Time |
|
Connectivity |
The agent's ability to successfully connect to the component. |
Float |
Percent |
|
Instances Saturation Percent |
The percentage of maximum available service instances in use. This metric is calculated by dividing the service's busy time with its maximum potential busy time, and multiplying this value by 100. The service busy time is calculated by multiplying the service's average response time ( The maximum potential busy time is calculated by multiplying the currently available service instances ( The following is an example:
Values approaching 100 percent may indicate that more instances are needed to handle incoming service requests. |
Float |
Percent |
|
Instances Used |
The total number of service instances in use. This metric is calculated by dividing the instances saturation percent metric value ( The following is an example:
|
Float |
Total |
|
Request Rate |
The average request rate of all ArcGIS Enterprise on Kubernetes services. |
Float |
Requests/second |
|
Request Response Time Avg |
The average response time of all ArcGIS Enterprise on Kubernetes services. |
Float |
Seconds |
|
Request Response Time Max |
The maximum response time of all ArcGIS Enterprise on Kubernetes services. |
Float |
Seconds |
|
Request Wait Time Avg |
The average time that all requests waited for an available service instance over the specified period. |
Float |
Seconds |
|
Request Wait Time Max |
The maximum time that a service request waited for an available service instance over the specified period. |
Float |
Seconds |
|
Request Wait Time Min |
The minimum time that a service request waited for an available service instance over the specified period. |
Float |
Seconds |
|
Requests Error Percent |
The percentage of failed requests of all ArcGIS Enterprise on Kubernetes services. |
Float |
Percent |
|
Requests Failed |
The total number of failed requests of all ArcGIS Enterprise on Kubernetes services. |
Float |
Total |
|
Requests Received |
The total number of received requests of all ArcGIS Enterprise on Kubernetes services. |
Float |
Total |
|
Services |
The total number of ArcGIS Enterprise on Kubernetes services. |
Float |
Total |
|
Pods |
The total number of pods. |
Float |
Pods |
|
Pods Running |
The total number of pods in the Running phase. |
Float |
Pods |
|
Pods Pending |
The total number of pods in the Pending phase. |
Float |
Pods |
|
Pods Failed |
The total number of pods in the Failed phase. |
Float |
Pods |
|
Pods Unknown |
The total number of pods in the Unknown phase. |
Float |
Pods |
ArcGIS Enterprise on Kubernetes services metrics
Monitor collects information about the following ArcGIS Server service types:
Feature
Geocode
Geodata
Geoprocessing
Image
Map
Network Analyst
All metrics in this table are collected for services with a dedicated instance type. For services with a shared or hosted instance type, Monitor collects only request and instance usage metrics; their pod and resource usage metrics are collected for the related shared service pool.
The following table lists the metrics that are collected for ArcGIS Enterprise on Kubernetes services:
|
Metric name |
Description |
Data type |
Unit |
|---|---|---|---|
|
Instances Saturation Percent |
The percentage of maximum configured service instances in use as derived by the number of running pods. Values approaching 100 percent may indicate that more instances are needed to handle incoming service requests. |
Float |
Percent |
|
Instances Used |
The total number of service instances in use. |
Float |
Total |
|
Request Rate |
The service's average request rate. |
Float |
Requests/second |
|
Request Response Time Avg |
The service's average response time. |
Float |
Seconds |
|
Request Response Time Max |
The service's maximum response time. |
Float |
Seconds |
|
Request Wait Time Avg |
The average time that service requests waited for an available service instance over the specified period. |
Float |
Seconds |
|
Request Wait Time Max |
The maximum time that a service request waited for an available service instance over the specified period. |
Float |
Seconds |
|
Request Wait Time Min |
The minimum time that a service request waited for an available service instance over the specified period. |
Float |
Seconds |
|
Requests Error Percent |
The percentage of failed service requests. |
Float |
Percent |
|
Requests Failed |
The total number of failed service requests. |
Float |
Total |
|
Requests Received |
The total number of received service requests. |
Float |
Total |
|
Pods |
The total number of pods for the service. |
Float |
Pods |
|
Pods Running |
The total number of pods for the service in the Running phase. |
Float |
Pods |
|
Pods Pending |
The total number of pods for the service in the Pending phase. |
Float |
Pods |
|
Pods Failed |
The total number of pods for the service in the Failed phase. |
Float |
Pods |
|
Pods Unknown |
The total number of pods for the service in the Unknown phase. |
Float |
Pods |
|
CPU Utilized |
The proportion of schedulable cluster CPU capacity in use by the service. |
Float |
Percent |
|
CPU Limit Utilized |
The proportion of the service's configured CPU limit in use. |
Float |
Percent |
|
Memory Utilized |
The proportion of schedulable cluster memory capacity in use by the service. |
Float |
Percent |
|
Memory Limit Utilized |
The proportion of the service's configured memory limit in use. |
Float |
Percent |
Web applications and web maps
The following table lists the metrics that are collected for web applications and web maps:
|
Metric name |
Description |
Data type |
Unit |
|---|---|---|---|
|
Active Sessions |
The total number of active sessions over the interval for the web application or web map |
Float |
Total |
|
Active Sessions Time |
The duration of all active sessions for the web application or web map over the interval |
Float |
Time |
Note:
Kubernetes API access is optional. Without it, Kubernetes-derived pod and resource utilization metrics and application usage sessions are not collected. Prometheus credentials are also optional; without them, service request metrics are not collected.
Component relationships
The inventory observer creates relationships between the ArcGIS Enterprise on Kubernetes component and its GIS services, web maps, and web applications. It also creates relationships from web maps and web applications to the services and web maps they use when the related items can be resolved.