Prometheus Server View
Prometheus with local data storage enabled scrapes the metrics of all hosts and components. The metrics are then reported to and stored on AOM or a third-party monitoring platform. The Prometheus Server view displays some built-in metrics provided by Prometheus, which can be used to monitor and measure system performance and status.
Metric Description
You can view Prometheus Server metrics in the following table.
Metric |
Unit |
Description |
---|---|---|
Target Sync |
second |
Collection latency of the target. |
Targets |
N/A |
Total number of metrics collected by the target. |
Average Scrape Interval |
second |
Interval for collecting metrics. |
Scrape Failures |
N/A |
The number of collection failures. |
Appended Samples Rate |
N/A |
The rate at which samples are added to the head. |
Head Series |
N/A |
The number of series in the head. |
Head Chunks |
N/A |
The number of head blocks. |
Query Rate |
N/A |
The number of Prometheus queries per second. |
P90 Query Duration |
second |
Duration for querying 90% of operations on different shards. |
Remote Sample Lag Ratio |
second |
Percentage of the highest timestamp of samples stored in WAL. |
Remote Write Traffic |
byte/s |
Remote write rate. |
Current Shards |
N/A |
The number of shards used for parallel sending to the remote storage. |
Max Shards |
N/A |
Maximum number of shards that the queue is allowed to run. |
Min Shards |
N/A |
Minimum number of shards that the queue is allowed to run. |
Desired Shards |
N/A |
The number of shards that the queue wants to run based on the percentage of input samples to output samples. |
Shard Capacity |
N/A |
Capacity of each shard of the queue used for parallel sending to the remote storage. |
Pending Samples |
N/A |
Capacity of each shard of the queue used for parallel sending to the remote storage. |
Current TSDB Segment |
N/A |
WAL segment index that TSDB is currently writing to. |
Current Segment of Remote Write |
N/A |
Current segment from which the WAL watcher is reading records. |
Sample Discard Rate |
N/A |
The rate at which samples are discarded after being read from WAL before being sent via remote write. |
Sample Failure Rate |
N/A |
Rate of samples that failed to be sent to remote storage due to unrecoverable errors. |
Sample Retry Rate |
N/A |
Rate of samples that failed to be sent to remote storage due to recoverable errors and were resent. |
Retry Rate of Enqueuing |
N/A |
Retry rate of enqueuing failed due to full shard queue. |
Metric List
The following is the metric list of the Prometheus Server view.
Metric |
Type |
Description |
---|---|---|
prometheus_target_sync_length_seconds_sum |
Summary |
Collection latencies of different targets. |
prometheus_sd_discovered_targets |
Gauge |
The number of metrics collected by different targets. |
prometheus_target_interval_length_seconds_sum |
Summary |
Collection interval. |
prometheus_target_scrapes_exceeded_body_size_limit_total |
Counter |
The number of collection failures. |
prometheus_tsdb_head_samples_appended_total |
Counter |
Total number of samples added to the head. |
prometheus_tsdb_head_series |
Gauge |
The number of series in the head block. |
prometheus_tsdb_head_chunks |
Gauge |
The number of chunks in the head block. |
prometheus_engine_query_duration_seconds_count |
Counter |
The number of queries. |
prometheus_engine_query_duration_seconds |
Counter |
Time taken by queries to complete, in seconds. |
prometheus_remote_storage_highest_timestamp_in_seconds |
Gauge |
Latest timestamp in the remote storage. |
prometheus_remote_storage_queue_highest_sent_timestamp_seconds |
Gauge |
Latest timestamp in the Prometheus shard. |
prometheus_remote_storage_bytes_total |
Counter |
Total number of bytes of data (non-metadata) sent by the queue after compression. |
prometheus_remote_storage_shards |
Gauge |
The number of shards used for parallel sending to the remote storage. |
prometheus_remote_storage_shards_max |
Gauge |
Maximum number of shards that the queue is allowed to run. |
prometheus_remote_storage_shards_min |
Gauge |
Minimum number of shards that the queue is allowed to run. |
prometheus_remote_storage_shards_desired |
Gauge |
The number of shards that the queue wants to run based on the percentage of input samples to output samples. |
prometheus_remote_storage_shard_capacity |
Gauge |
Capacity of each shard of the queue used for parallel sending to the remote storage. |
prometheus_remote_storage_pending_samples |
Gauge |
The number of pending samples in queue shards to be sent to remote storage. |
prometheus_tsdb_wal_segment_current |
Gauge |
WAL segment index that TSDB is currently writing to. |
prometheus_wal_watcher_current_segment |
Gauge |
Current segment from which the WAL is reading records. |
prometheus_remote_storage_dropped_samples_total |
Gauge |
The rate at which samples are discarded after being read from WAL before being sent via remote write. |
prometheus_remote_storage_failed_samples_total |
Gauge |
Rate of samples that failed to be sent to remote storage due to unrecoverable errors. |
prometheus_remote_storage_retried_samples_total |
Gauge |
The number of samples that failed to be sent to remote storage due to recoverable errors and were resent. |
prometheus_remote_storage_enqueue_retries_total |
Gauge |
The number of retries upon enqueuing failed due to full shard queue. |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.