Menu

Labels vs annotations: how one YAML indent made Prometheus scrape Pyroscope twice
📰

Labels vs annotations: how one YAML indent made Prometheus scrape Pyroscope twice

DEV Community: kubernetes·Ali Ahmed·11 days ago
#yIhsXtwx
#devtokubernetes#devto#dev#service#prometheus
Reading 0:00
15s threshold

A Prometheus ServiceMonitor selects Services by labels . It never looks at annotations . In Grafana Pyroscope's Helm chart, a refactor nested prometheus.io/service-monitor: "false" under annotations instead of labels on the headless Service, so the exclusion silently stopped working. Prometheus scraped every pod twice, as job="pyroscope" and job="pyroscope-headless" . My fix, grafana/pyroscope#5377 (in review), moves the key back under metadata.labels .

What's the difference between labels and annotations in Kubernetes?

Both are key-value metadata, but they do different jobs:

Aspect Labels Annotations

Purpose Identify and select objects Attach non-identifying information

Used by selectors (Services, ServiceMonitors, kubectl -l ) Yes No

Typical content app , component , tier build info, tool configuration, long text

Some tools read annotations for configuration, like the old prometheus.io/scrape convention for annotation-based scraping. That's exactly why the two get mixed up.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.