Prometheus Server View
Prometheus with local data storage enabled scrapes the metrics of all hosts and components. The metrics are then reported to and stored on AOM or a third-party monitoring platform. The Prometheus Server view displays some built-in metrics provided by Prometheus, which can be used to monitor and measure system performance and status.
Metric Description
You can view Prometheus Server metrics in the following table.
Metric | Unit | Description |
|---|---|---|
Target Sync | second | Collection latency of the target. |
Targets | N/A | Total number of metrics collected by the target. |
Average Scrape Interval | second | Interval for collecting metrics. |
Scrape Failures | N/A | The number of collection failures. |
Appended Samples Rate | N/A | The rate at which samples are added to the head. |
Head Series | N/A | The number of series in the head. |
Head Chunks | N/A | The number of head blocks. |
Query Rate | N/A | The number of Prometheus queries per second. |
P90 Query Duration | second | Duration for querying 90% of operations on different shards. |
Remote Sample Lag Ratio | second | Percentage of the highest timestamp of samples stored in WAL. |
Remote Write Traffic | byte/s | Remote write rate. |
Current Shards | N/A | The number of shards used for parallel sending to the remote storage. |
Max Shards | N/A | Maximum number of shards that the queue is allowed to run. |
Min Shards | N/A | Minimum number of shards that the queue is allowed to run. |
Desired Shards | N/A | The number of shards that the queue wants to run based on the percentage of input samples to output samples. |
Shard Capacity | N/A | Capacity of each shard of the queue used for parallel sending to the remote storage. |
Pending Samples | N/A | Capacity of each shard of the queue used for parallel sending to the remote storage. |
Current TSDB Segment | N/A | WAL segment index that TSDB is currently writing to. |
Current Segment of Remote Write | N/A | Current segment from which the WAL watcher is reading records. |
Sample Discard Rate | N/A | The rate at which samples are discarded after being read from WAL before being sent via remote write. |
Sample Failure Rate | N/A | Rate of samples that failed to be sent to remote storage due to unrecoverable errors. |
Sample Retry Rate | N/A | Rate of samples that failed to be sent to remote storage due to recoverable errors and were resent. |
Retry Rate of Enqueuing | N/A | Retry rate of enqueuing failed due to full shard queue. |
Metric List
The following is the metric list of the Prometheus Server view.
Metric | Type | Description |
|---|---|---|
prometheus_target_sync_length_seconds_sum | Summary | Collection latencies of different targets. |
prometheus_sd_discovered_targets | Gauge | The number of metrics collected by different targets. |
prometheus_target_interval_length_seconds_sum | Summary | Collection interval. |
prometheus_target_scrapes_exceeded_body_size_limit_total | Counter | The number of collection failures. |
prometheus_tsdb_head_samples_appended_total | Counter | Total number of samples added to the head. |
prometheus_tsdb_head_series | Gauge | The number of series in the head block. |
prometheus_tsdb_head_chunks | Gauge | The number of chunks in the head block. |
prometheus_engine_query_duration_seconds_count | Counter | The number of queries. |
prometheus_engine_query_duration_seconds | Counter | Time taken by queries to complete, in seconds. |
prometheus_remote_storage_highest_timestamp_in_seconds | Gauge | Latest timestamp in the remote storage. |
prometheus_remote_storage_queue_highest_sent_timestamp_seconds | Gauge | Latest timestamp in the Prometheus shard. |
prometheus_remote_storage_bytes_total | Counter | Total number of bytes of data (non-metadata) sent by the queue after compression. |
prometheus_remote_storage_shards | Gauge | The number of shards used for parallel sending to the remote storage. |
prometheus_remote_storage_shards_max | Gauge | Maximum number of shards that the queue is allowed to run. |
prometheus_remote_storage_shards_min | Gauge | Minimum number of shards that the queue is allowed to run. |
prometheus_remote_storage_shards_desired | Gauge | The number of shards that the queue wants to run based on the percentage of input samples to output samples. |
prometheus_remote_storage_shard_capacity | Gauge | Capacity of each shard of the queue used for parallel sending to the remote storage. |
prometheus_remote_storage_pending_samples | Gauge | The number of pending samples in queue shards to be sent to remote storage. |
prometheus_tsdb_wal_segment_current | Gauge | WAL segment index that TSDB is currently writing to. |
prometheus_wal_watcher_current_segment | Gauge | Current segment from which the WAL is reading records. |
prometheus_remote_storage_dropped_samples_total | Gauge | The rate at which samples are discarded after being read from WAL before being sent via remote write. |
prometheus_remote_storage_failed_samples_total | Gauge | Rate of samples that failed to be sent to remote storage due to unrecoverable errors. |
prometheus_remote_storage_retried_samples_total | Gauge | The number of samples that failed to be sent to remote storage due to recoverable errors and were resent. |
prometheus_remote_storage_enqueue_retries_total | Gauge | The number of retries upon enqueuing failed due to full shard queue. |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
