Grafana shows no data: reduce the query before changing collection

Problem

Grafana panel shows no data, but the metric exists in Prometheus.

Why it happens

The issue may be caused by wrong label filters, dashboard variables, namespace filters, datasource selection, or time range.

Recommended approach

Prove the metric exists with a raw PromQL query, then rebuild the Grafana query one constraint at a time. Add label filters, variables, aggregation, and transformations only after the simpler query returns data.

Debugging order

Useful checks:

• Confirm the panel uses the intended datasource
• Query the metric directly in Prometheus
• Match the panel time range and step
• Remove dashboard variables and label filters temporarily
• Reintroduce namespace, workload, and other filters one at a time
• Inspect Grafana transformations after the query returns data

Use this when

• Dashboard debugging
• Kubernetes monitoring
• Prometheus or Grafana troubleshooting

Avoid this when

• When Prometheus itself does not have the metric

Start with the smallest PromQL query

Open Prometheus or Grafana Explore and query only the metric name:

http_requests_total

Inspect the labels returned by a real series before adding selectors:

http_requests_total{namespace="production"}

If the first query works and the second does not, the problem is the label value or label name—not collection.

Debug a rate query

Counters normally need rate() or increase(), but the range must contain enough samples:

rate(http_requests_total[$__rate_interval])

If the scrape interval is 60 seconds, a hard-coded [1m] range can contain too few points. Configure the Grafana Prometheus datasource with the real scrape interval and prefer $__rate_interval for dashboard rate queries.

Inspect the exact request sent by Grafana

In the panel editor, open Query inspector and check:

• The expanded PromQL after variables were substituted
• Start and end timestamps
• Step interval
• Datasource UID
• Raw response and error message

This catches variables that resolve to an empty string, All values that produce an invalid matcher, and dashboards using a different Prometheus datasource than Explore.

Common causes and checks

Wrong variable matcher

Multi-value variables generally require a regex matcher:

http_requests_total{namespace=~"$namespace"}

Using namespace="$namespace" can fail when the variable expands to several values or a regex-style All value.

Time range excludes the samples

Switch temporarily to a wider range and use an instant query for gauges. Check whether the source stopped scraping recently.

Transformation removes the result

Disable panel transformations, overrides, and value mappings until the raw query result is visible. Reintroduce them one at a time.

Aggregation uses a missing label

Inspect actual labels before grouping:

sum by (namespace, pod) (rate(http_requests_total[$__rate_interval]))

If the exporter uses different labels, the aggregation may collapse or rename the series in an unexpected way.

References

• Grafana Prometheus query inspector
• Troubleshoot Prometheus datasource issues