Prometheus target down: debug context deadline exceeded
Diagnose Prometheus targets showing context deadline exceeded by checking the rendered scrape URL, DNS, network policy, TLS, authentication, and exporter latency.
Published
Problem
Prometheus shows a target as DOWN with context deadline exceeded. The message means the scrape did not complete within its timeout; it does not identify whether DNS, connection establishment, TLS, authentication, or a slow /metrics response consumed the time.
Quick answer
Open Status → Targets and record the exact scrape URL, last error, labels, and scrape pool. Query the generated health metric:
promql
up{job="<job-name>"}Then test the same scheme, host, port, and path from the Prometheus network context—not from your laptop:
bash
curl --verbose --max-time 10 http://<target>:<port>/<metrics-path>Use an ephemeral debug Pod or approved diagnostic container when the Prometheus image has no shell. Do not permanently add troubleshooting tools to the production image.
Validate discovery and relabeling
Prometheus may discover the expected Pod but produce a different final address after relabeling. The Targets page and /api/v1/targets distinguish discovered labels from final labels:
bash
curl -s http://<prometheus>:9090/api/v1/targetsConfirm namespace selectors, labels, port name, scheme, metrics path, and final address. For Prometheus Operator resources, inspect the rendered ServiceMonitor or PodMonitor selection before changing exporter configuration.
Separate timeout stages
DNS or connection timeout
Check cluster DNS, Service endpoints, security groups, firewall rules, routing, and Kubernetes NetworkPolicy. A Service with no ready endpoints accepts discovery but cannot route to a healthy exporter.
TLS handshake failure
The last error often names an unknown authority, hostname mismatch, or protocol error. Ensure scheme: https, CA trust, server name, and certificate SANs agree. Avoid insecureSkipVerify as a permanent fix.
Authentication failure
Authentication normally returns 401 or 403, but a proxy or external identity service can time out. Verify the selected Secret, token mount, and authorization policy without exposing credentials.
Exporter is too slow
Measure the endpoint response time and inspect exporter logs. Database or cloud exporters can block on their upstream dependency. Increasing scrape_timeout hides the symptom unless the expected collection duration is genuinely longer and remains below the scrape interval.
Useful checks
bash
kubectl get service,endpoints,endpointslice -n <namespace>
kubectl get networkpolicy -n <namespace>
kubectl logs <exporter-pod> -n <namespace>If the target is UP but the dashboard is empty, follow Grafana shows no data. If the target never appears, use PodMonitor exists but Prometheus does not scrape.