Monitoring

The Union.ai data plane runs its own Prometheus instance, which collects the metrics that platform features rely on: cost tracking, task-level resource monitoring, and execution observability. It is pre-configured and needs no setup.

For operational monitoring of the cluster itself (node health, API server metrics, CoreDNS, and so on), the data plane chart also includes an optional kube-prometheus-stack, which you turn on separately.

Union.ai features Prometheus

The chart deploys the community prometheus chart as a subchart, with kube-state-metrics, in the release namespace. It needs no Prometheus Operator and no CRDs: the chart writes the scrape configuration and recording rules into the union-prometheus ConfigMap. This instance is a platform dependency. Don’t replace it or change its scrape configuration.

Other data plane components reach it at:

http://union-operator-prometheus.<NAMESPACE>.svc/prometheus

Scrape targets

Recording rules

When cost.enabled is true, the default, the chart loads its cost and GPU recording rules into this Prometheus.

Metrics sent to the control plane

In zero-trust mode (zero_trust.enabled, the default), Prometheus also forwards the metrics it collects to the control plane, with Prometheus remote write to https://<CONTROLPLANE_HOST>/metrics/remote_write. It authenticates with OAuth2 as the cluster’s app, secrets.admin.clientId, using the client secret in the union-secret-auth secret, so the chart requires secrets.admin.enable: true in this mode. With zero_trust.enabled: false, nothing is forwarded.

RBAC

Which permissions Prometheus has depends on the RBAC mode:

• Low-privilege mode (the chart default). The chart creates a namespace-scoped Role for Prometheus and kube-state-metrics. kube-state-metrics reports only the release namespace, so node-level metrics (kube_node_*) are missing, and Prometheus can’t scrape cAdvisor, so task-level CPU and memory charts show no data.

• Standard mode (low_privilege: false). The chart creates no RBAC for Prometheus. Turn on the subcharts’ own ClusterRoles, or Prometheus can’t scrape anything:

  prometheus:
    rbac:
      create: true
    kube-state-metrics:
      rbac:
        create: true

A data plane connected from the Union.ai UI runs in standard mode with these ClusterRoles already on.

Configuration

Configure the instance under prometheus.server, using the community chart’s values. The chart’s defaults:

prometheus:
  server:
    retention: "3d"
    persistentVolume:
      enabled: false
    emptyDir:
      sizeLimit: "10Gi"
    resources:
      limits:
        cpu: "3"
        memory: "3500Mi"
      requests:
        cpu: "1"
        memory: "1Gi"

Cluster health monitoring

To turn on operational monitoring with the Prometheus Operator, Grafana, and node-exporter:

monitoring:
  enabled: true

This deploys kube-prometheus-stack with these defaults:

• Prometheus with 7-day retention
• Grafana, with the admin password admin: set monitoring.grafana.adminPassword in production
• Node exporter, kubelet, kube-state-metrics, API server and CoreDNS monitoring. Etcd, scheduler, controller manager and kube-proxy monitoring are off.
• Default alerting and recording rules. Alertmanager is off.

Prometheus Operator CRDs

The stack’s Prometheus Operator discovers scrape targets and rules through CRDs (ServiceMonitor, PodMonitor, PrometheusRule, and others). The chart doesn’t install them (monitoring.crds.enabled: false), so apply them yourself, with server-side apply, from the helm-charts repository at your chart version:

git clone --depth 1 --branch dataplane-<CHART_VERSION> https://github.com/unionai/helm-charts.git
kubectl apply --server-side --force-conflicts -f helm-charts/crds/kube-prometheus-stack/

Union.ai monitoring resources

When the Prometheus Operator CRDs are installed, by this stack or by a Prometheus Operator you already run, the chart also creates these resources in the release namespace, each labeled release: <RELEASE>. The PrometheusRules are named after the release, for example union-dataplane-dataplane-rules for a release named union:

The operational rules cover the health of the Union.ai services and pod scheduling. union:dp:pods_unschedulable:sum counts the task pods that can’t be scheduled, per namespace, and the UnionDPPodsUnschedulable alert fires when a namespace has pods stuck Pending for 15 minutes, for example on insufficient node capacity or a taint the pods don’t tolerate. It’s the main signal for “why are my task pods Pending?”. It reads kube_pod_status_unschedulable, which the stack’s kube-state-metrics collects but the Union.ai features instance drops.

The chart also ships Grafana dashboards as ConfigMaps labeled grafana_dashboard: "1" (monitoring.dashboards.enabled, on), which the Grafana sidecar picks up.

Customizing the monitoring stack

The monitoring key accepts all kube-prometheus-stack values. Common overrides:

monitoring:
  enabled: true

  # Grafana
  grafana:
    enabled: true
    adminPassword: "my-secure-password"
    ingress:
      enabled: true
      ingressClassName: nginx
      hosts:
        - grafana.example.com

  # Prometheus retention and resources
  prometheus:
    prometheusSpec:
      retention: 30d
      resources:
        requests:
          memory: "2Gi"

  # Alertmanager
  alertmanager:
    enabled: true
    # Configure receivers, routes, etc.

The stack’s Prometheus supports remote write for forwarding metrics to an external time-series database, such as Amazon Managed Prometheus, Grafana Cloud, or Thanos:

monitoring:
  prometheus:
    prometheusSpec:
      remoteWrite:
        - url: "https://aps-workspaces.<REGION>.amazonaws.com/workspaces/<WORKSPACE_ID>/api/v1/remote_write"
          sigv4:
            region: <REGION>

Scraping Union.ai services from your own Prometheus

If you already run Prometheus, you can scrape the Union.ai services for operational visibility. This adds to the built-in instance; it doesn’t replace it.

With the Prometheus Operator, you don’t need to write anything: the chart’s union-service-monitor ServiceMonitor (see Union.ai monitoring resources) selects the services. Make sure your Prometheus selects ServiceMonitors in the data plane namespace, or with the label release: <RELEASE>.

Without it, add a static scrape job:

scrape_configs:
  - job_name: union-dataplane-services
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names: [<NAMESPACE>]
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_platform_union_ai_prometheus_group]
        regex: union-services
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        regex: debug
        action: keep

Further reading

• Prometheus documentation: configuration, querying, and operation
• Prometheus remote write: forwarding metrics to external storage
• Prometheus kubernetes_sd_config: Kubernetes service discovery for scrape targets
• kube-prometheus-stack chart: the full monitoring stack with Grafana and alerting