Infra Notes

PodMonitor exists but the target is missing: trace selector discovery

Debug a missing Prometheus target by following the selector chain from Prometheus to PodMonitor, pod labels, and the named metrics port.

Published · Updated

KubernetesPrometheusPodMonitorObservabilityDebugging

Problem

PodMonitor is created, but Prometheus does not scrape the target.

Why it happens

Common causes include label selector mismatch, namespace selector mismatch, wrong port name, missing pod labels, or Prometheus not selecting that PodMonitor.

Recommended approach

Trace discovery in order: confirm Prometheus selects the PodMonitor, the PodMonitor selects the namespace and pods, and its endpoint references the pod container port by the correct name. Only debug the application after the target is discovered.

Useful checks

Useful checks:

bash

kubectl get podmonitor -A
kubectl get pods --show-labels
kubectl describe podmonitor <name>
kubectl get pods -n <namespace> -o yaml
kubectl port-forward svc/prometheus 9090:9090

In the Prometheus UI, inspect Status → Service Discovery and Targets. A dropped target usually exposes the label or selector mismatch more directly than the Kubernetes object alone.

Use this when

  • When the metrics endpoint works but the target is missing in Prometheus

Avoid this when

  • When the application itself does not expose metrics

How PodMonitor discovery works

A successful scrape requires every selector in this chain to match:

text

Prometheus
  -> selects the PodMonitor object
  -> PodMonitor selects a namespace
  -> PodMonitor selector matches pod labels
  -> pod exposes the named container port
  -> metrics path responds successfully

Do not jump directly to changing the exporter. Find the first link in this chain that returns no object.

1. Confirm Prometheus selects the PodMonitor

Inspect the Prometheus custom resource:

bash

kubectl get prometheus -A
kubectl get prometheus <prometheus-name> -n <monitoring-namespace> -o yaml
kubectl get podmonitor <monitor-name> -n <monitor-namespace> --show-labels

Compare the PodMonitor labels with spec.podMonitorSelector. Also check spec.podMonitorNamespaceSelector; a PodMonitor in another namespace is invisible unless the namespace selector permits it.

2. Confirm the PodMonitor selects the pods

bash

kubectl get podmonitor <monitor-name> -n <namespace> -o yaml
kubectl get pods -n <target-namespace> -l '<label-selector>' --show-labels

The PodMonitor's spec.selector matches pod labels, not Service labels. If the command returns no pods, correct the selector or workload labels.

3. Verify the named metrics port

For a PodMonitor endpoint, port normally refers to the name of a container port:

yaml

spec:
  podMetricsEndpoints:
    - port: metrics
      path: /metrics

The pod must expose the same name:

yaml

ports:
  - name: metrics
    containerPort: 8080

A numeric container port with no matching name is a common reason the target is dropped.

4. Test the endpoint from inside the cluster

bash

kubectl port-forward -n <namespace> pod/<pod-name> 8080:<metrics-port>
curl -fsS http://127.0.0.1:8080/metrics | head

If discovery works but the target is DOWN, check the path, scheme, TLS settings, authentication, network policy, and whether the process listens on the pod IP instead of only 127.0.0.1.

Interpret the Prometheus UI

  • Missing from Service Discovery: Prometheus did not select the PodMonitor
  • Present but dropped: a namespace, label, relabeling, or port selector did not match
  • Present in Targets but DOWN: discovery succeeded; inspect the scrape error
  • UP but queries return nothing: inspect metric names and labels instead

References