A Prometheus ServiceMonitor selects Services by labels . It never looks at annotations . In Grafana Pyroscope's Helm chart, a refactor nested prometheus.io/service-monitor: "false" under annotations instead of labels on the headless Service, so the exclusion silently stopped working. Prometheus scraped every pod twice, as job="pyroscope" and job="pyroscope-headless" . My fix, grafana/pyroscope#5377 (in review), moves the key back under metadata.labels .
What's the difference between labels and annotations in Kubernetes?
Both are key-value metadata, but they do different jobs:
Aspect Labels Annotations
Purpose Identify and select objects Attach non-identifying information
Used by selectors (Services, ServiceMonitors, kubectl -l ) Yes No
Typical content app , component , tier build info, tool configuration, long text
Some tools read annotations for configuration, like the old prometheus.io/scrape convention for annotation-based scraping. That's exactly why the two get mixed up.…