Prometheus matters not only as a de facto standard, but as a legible monitoring architecture where the strengths and limits of pull-based collection are easy to see.
In real operations, this chapter helps treat scrape jobs, targets, service discovery, recording rules, and alerting rules as one connected system instead of a pile of unrelated YAML fragments.
In interviews and engineering discussions, it helps explain why Prometheus is great for baseline cloud-native monitoring yet does not solve long retention or global scale by itself.
Practical value of this chapter
Scrape topology
Design jobs, targets, and service discovery to prevent monitoring blind spots and duplicate time series.
Rules and alert pipeline
Separate recording and alerting rules to stabilize dashboard latency and improve alert quality.
Remote write boundary
Define local Prometheus versus long-term storage responsibilities by SLA and cost constraints.
Interview articulation
Explain pull-model trade-offs and when an additional aggregation layer becomes necessary.
Decision frame and editorial focus
Chapter focus
Prometheus, metric scraping, rules, and monitoring reliability
Workload profile
Start from the specialized query: analytics, search, time series, graph traversal, vector retrieval, or monitoring metrics.
Good fit
The choice is justified when the index or storage model directly matches product behavior and relieves the source of truth.
Boundary and risk
The danger is turning a specialized layer into a universal database and losing consistency, freshness, and ownership boundaries.
Connect next
Connect the chapter to the OLTP source, data pipeline, retention/compaction, and read-model architecture.
Source
Prometheus docs
The Prometheus model from the primary source: pull scraping, TSDB, PromQL, and rules — no second-hand retelling.
When you need to know what the infrastructure is doing right now, the choice almost always lands on Prometheus — a purpose-built monitoring time-series system that combines pull-based metric collection, its own TSDB engine, and PromQL query semantics. On the TSDB map it is the canonical tool for infrastructure monitoring and SLO-driven operations.
History: key milestones
Born at SoundCloud
Prometheus started as an internal monitoring engine for cloud-native workloads.
Open source and early ecosystem
The project became open source and quickly gained adoption in Kubernetes environments.
CNCF incubating stage
Prometheus joined CNCF and established a neutral governance model.
CNCF graduated
It reached graduated status and became an industry standard for infrastructure monitoring.
Prometheus 2.x as the production baseline
Remote write/read stabilized, and the operators and practices around the project learned to keep label cardinality inside safe bounds.
Evolution toward scalable metric-storage profiles
A single Prometheus node hits the local disk sooner than you would like — hence the mature patterns for long-term storage, federation, and hybrid monitoring architectures.
Prometheus specifics
Pull-based collection
Prometheus goes after metrics itself. The source of truth about which targets are scraped and whether they are alive stays on the monitoring side, not scattered across applications.
TSDB with WAL + blocks
Fresh samples first live in memory and the WAL, then compact into blocks on disk. The cost of that choice is the local disk as a ceiling: long-term storage has to move outside.
PromQL as the query language
PromQL is built for time-series vectors, label-based aggregation, and time-window analysis. The same language drives both dashboards and alert conditions — no separate language for alerts.
Rule-driven alerting
Recording rules, alerting rules, and Alertmanager close the response loop. The weak spot shows up when there are too many rules and the signal drowns in noise.
Prometheus architecture by layers
At a high level, Prometheus can be read as a pipeline: target scraping -> TSDB head/WAL -> block storage -> PromQL engine -> rules and alerts -> external integrations.
Key features
Prometheus is optimized for monitoring workloads: target scraping, WAL/block-based TSDB, PromQL, and rule-driven alerting.
Pull model
Label model
Data lifecycle
Configuration and data: DDL/DML analogy
Prometheus does not implement SQL DDL/DML literally. But for reasoning about the architecture it helps to keep them apart: DDL-like operations (scrape/rule topology updates) are one thing, and DML-like operations (metric sample flow and PromQL read execution) are another.
How the DDL/DML model works in Prometheus
DDL-like: scrape/rule topology updates. DML-like: sample flow and PromQL reads.
1. Scrape / ingest
Samples + queriesScraper or remote write ingest receives new metric samples.
2. WAL append
Samples + queriesSamples are appended to WAL for durability before deeper processing.
3. Head update
Samples + queriesTSDB head updates series state and label index for fresh data.
4. PromQL execution
Samples + queriesQuery engine reads head and historical blocks, then aggregates results.
5. Compaction + retention
Samples + queriesBackground compaction merges blocks and retention removes expired data.
Active step
1. Scrape / ingest
Scraper or remote write ingest receives new metric samples.
Data and query path
- The DML-like path covers ingest, storage, and PromQL read execution.
- Fresh data lives in head, historical data in compacted blocks.
- Label cardinality has a direct impact on cost and query latency.
Source
InfluxDB docs
Reference context for an alternative TSDB profile.
Prometheus vs InfluxDB
Core approach
Prometheus: Pull-based target scraping with tight integration into monitoring workflows.
InfluxDB: Strong focus on ingest APIs and time-series storage for broad telemetry scenarios.
Query language
Prometheus: PromQL focused on metrics, labels, and alert-oriented analysis.
InfluxDB: InfluxQL/Flux depending on version and data-processing profile.
Typical production profile
Prometheus: Infrastructure monitoring, SLOs, alerting, and Kubernetes observability.
InfluxDB: Monitoring, IoT, and telemetry where flexible ingestion and retention policies are key.
Operating model
Prometheus: A single node holds fresh data, and federation or remote storage is added on top for long-term retention.
InfluxDB: Often runs as a standalone TSDB layer or a managed service, without a separate long-term retention tier.
Why Prometheus is often chosen for monitoring
What this actually buys you at the system-design stage:
- A simple pull model and a dense Kubernetes ecosystem made Prometheus the cloud-native monitoring standard — it is easier to wire in than to argue for an alternative.
- PromQL and rules keep observability and alerting in one loop: there is no separate DSL to stand up for alerts.
- The WAL -> active head -> blocks lifecycle keeps metric writes and reads predictable under load, until you hit the local disk.
- Integration with Alertmanager, Grafana, and remote storage maps the route from a single node to a scalable topology — you can grow gradually instead of rewriting everything at once.
PromQL query examples
A compact cheat sheet for common production tasks: load, latency, error control, and SLO monitoring.
Throughput (RPS) by service
Estimate incoming traffic for `checkout-api` over the last 5 minutes.
sum(rate(http_requests_total{service="checkout-api"}[5m]))A baseline signal for incoming traffic speed, usually one of the first RED dashboard panels.
P95 latency
Track latency degradation in the user-facing request path.
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="checkout-api"}[5m])) by (le))For p95/p99, use histogram `*_bucket` data instead of averages to capture tail behavior.
Error rate (%)
Measure the share of 5xx responses relative to total service traffic.
100 * (sum(rate(http_requests_total{service="checkout-api",status=~"5.."}[5m])) / sum(rate(http_requests_total{service="checkout-api"}[5m])))A common base metric for SLO-aligned alerting and burn-rate rules.
CPU saturation by pod
Find pods that are creeping toward their CPU limits.
sum(rate(container_cpu_usage_seconds_total{namespace="prod",pod=~"checkout-api-.*"}[5m])) by (pod)This is where you see when autoscaling needs tuning and which hot pod replica is dragging latency up.
Top-k by memory usage
Quickly isolate the heaviest pods by working set memory.
topk(5, container_memory_working_set_bytes{namespace="prod",pod=~"checkout-api-.*"})Helpful for OOMKill investigations and requests/limits right-sizing.
Error budget burn rate
Estimate how quickly the service is spending its error budget for a 99.9% SLO.
(sum(rate(http_requests_total{service="checkout-api",status=~"5.."}[5m])) / sum(rate(http_requests_total{service="checkout-api"}[5m]))) / (1 - 0.999)Values significantly above `1` indicate the service is burning budget faster than allowed.
References
Related chapters
- Time Series Databases (TSDB): types, trade-offs, and selection - How to position Prometheus against alternative TSDB families across latency, retention, and operating profile.
- Database Selection Framework - Selection framework that helps justify Prometheus as a monitoring-focused TSDB rather than a universal analytical datastore.
- Observability & Monitoring Design - Prometheus metrics alone do not give the full picture: this chapter ties them to logs, traces, and SLO/SLI loops in production observability architecture.
- Prometheus: The Documentary - Historical context for Prometheus evolution and why its ecosystem became central in cloud-native monitoring.
- Kubernetes Fundamentals - Foundational context for service discovery, pull scraping, and operator-based rollout patterns in Kubernetes.
- Service Discovery - If target discovery misses new pods, gaps appear in your metrics exactly where they are hardest to notice — this chapter is about avoiding that.
- VictoriaMetrics: history and architecture - Comparison of Prometheus-compatible storage and scaling strategies for long-term retention workloads.