Metrics and Alerts

Table of Contents

MinIO Console Metrics Dashboard
Available Metrics

MinIO publishes cluster and node metrics using the Prometheus Data Model. You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting.

MinIO provides a scraping endpoint for cluster-level metrics:

http://minio.example.net:9000/minio/v2/metrics/cluster

Replace http://minio.example.net with the hostname of any node in the MinIO deployment. For deployments with a load balancer managing connections between MinIO nodes, specify the address of the load balancer.

MinIO by default requires authentication for scraping the metrics endpoints. Use the mc admin prometheus generate command to generate the necessary bearer tokens. You can alternatively disable metrics endpoint authentication by setting MINIO_PROMETHEUS_AUTH_TYPE to public.

MinIO Console Metrics Dashboard

The MinIO Console provides a point-in-time metrics dashboard by default:

MinIO Console with Point-In-Time Metrics

The Console also supports displaying time-series and historical data by querying a Prometheus service configured to scrape data from the MinIO deployment. Specifically, the MinIO Console uses Prometheus query API to retrieve stored metrics data and display the following visualizations:

Usage - provides historical and on-demand visualization of overall usage and status
Traffic - provides historical and on-demand visualization of network traffic
Resources - provides historical and on-demand visualization of resources (compute and storage)
Info - provides point-in-time status of the deployment

MinIO Console displaying Prometheus-backed Monitoring Data

To enable historical data visualization in MinIO Console, set the following environment variables on each node in the MinIO deployment:

Set MINIO_PROMETHEUS_URL to the URL of the Prometheus service
Set MINIO_PROMETHEUS_JOB_ID to the unique job ID assigned to the collected metrics

MinIO also publishes a Grafana Dashboard for visualizing collected metrics. For more complete documentation on configuring a Prometheus-compatible data source for Grafana, see Grafana Support for Prometheus.

Available Metrics

MinIO publishes the following metrics, where each metric includes a label for the MinIO server which generated that metric.

Object and Bucket Metrics

minio_bucket_objects_size_distribution: Distribution of object sizes in a given bucket. You can identify the bucket using the { bucket="STRING" } label.

minio_bucket_usage_object_total: Total number of objects in a given bucket. You can identify the bucket using the { bucket="STRING" } label.

minio_bucket_usage_total_bytes: Total bucket size in bytes in a given bucket. You can identify the bucket using the { bucket="STRING" } label.

minio_bucket_quota_total_bytes: Total bucket quota size in bytes. You can identify the bucket using the { bucket="STRING" } label.

Replication Metrics

These metrics are only populated for MinIO clusters with Server-Side Bucket Replication enabled.

minio_bucket_replication_failed_bytes: Total number of bytes that failed at least once to replicate for a given bucket. You can identify the bucket using the { bucket="STRING" } label

minio_bucket_replication_latency: Replication latency in milliseconds.

minio_bucket_replication_pending_bytes: Total number of bytes pending to replicate for a given bucket. You can identify the bucket using the { bucket="STRING" } label

minio_bucket_replication_received_bytes: Total number of bytes replicated to this bucket from another source bucket. You can identify the bucket using the { bucket="STRING" } label.

minio_bucket_replication_sent_bytes: Total number of bytes replicated to the target bucket. You can identify the bucket using the { bucket="STRING" } label.

minio_bucket_replication_pending_count: Total number of replication operations pending for a given bucket. You can identify the bucket using the { bucket="STRING" } label.

minio_bucket_replication_failed_count: Total number of replication operations failed for a given bucket. You can identify the bucket using the { bucket="STRING" } label.

Capacity Metrics

minio_cluster_capacity_raw_free_bytes: Total free capacity online in the cluster.

minio_cluster_capacity_raw_total_bytes: Total capacity online in the cluster.

minio_cluster_capacity_usable_free_bytes: Total free usable capacity online in the cluster.

minio_cluster_capacity_usable_total_bytes: Total usable capacity online in the cluster.

minio_node_disk_free_bytes: Total storage available on a specific drive for a node in the MinIO deployment. You can identify the drive and node using the { disk="/path/to/disk",server="STRING"} labels respectively.

minio_node_disk_total_bytes: Total storage on a specific drive for a node in the MinIO deployment. You can identify the drive and node using the { disk="/path/to/disk",server="STRING"} labels respectively.

minio_node_disk_used_bytes: Total storage used on a specific drive for a node in a MinIO deployment. You can identify the drive and node using the { disk="/path/to/disk",server="STRING"} labels respectively.

Lifecycle Management Metrics

minio_cluster_ilm_transitioned_bytes: Total number of bytes transitioned using tiering/transition lifecycle management rules

minio_cluster_ilm_transitioned_objects: Total number of objects transitioned using tiering/transition lifecycle management rules

minio_cluster_ilm_transitioned_versions: Total number of non-current object versions transitioned using tiering/transition lifecycle management rules

minio_node_ilm_transition_pending_tasks: Total number of pending object transition tasks

minio_node_ilm_transition_active_tasks: Number of active ILM transition tasks

minio_node_ilm_expiry_pending_tasks: Total number of pending object expiration tasks

minio_node_ilm_expiry_active_tasks: Total number of active object expiration tasks

minio_node_ilm_versions_scanned: Total number of object versions checked for ilm actions since server start

Node and Drive Health Metrics

minio_cluster_disk_online_total: The total number of drives online

minio_cluster_disk_offline_total: The total number of drives offline

minio_cluster_disk_total: The total number of drives

minio_cluster_nodes_offline_total: Total number of MinIO nodes offline.

minio_cluster_nodes_online_total: Total number of MinIO nodes online.

minio_node_disk_free_inodes: Total free inodes.

minio_node_disk_latency_us: Average last minute latency in µs for drive API storage operations.

minio_node_disk_offline_total: Total drives offline.

minio_node_disk_online_total: Total drives online.

minio_node_disk_total: Total drives.

minio_heal_objects_errors_total: Objects for which healing failed in current self healing run

minio_heal_objects_heal_total: Objects healed in current self healing run

minio_heal_objects_total: Objects scanned in current self healing run

minio_heal_time_last_activity_nano_seconds: Time elapsed (in nano seconds) since last self healing activity. This is set to -1 until initial self heal

Notification Queue Metrics

minio_audit_target_queue_length: Total number of unsent audit messages in the queue.

minio_audit_total_messages: Total number of audit messages sent since last server start.

minio_audit_failed_messages: Total number of audit messages which failed to send since last server start.

minio_notify_current_send_in_progress: Total number of notification messages in progress to configured targets.

minio_notify_target_queue_length: Total number of unsent notification messages in the queue.

Scanner Metrics

minio_node_scanner_bucket_scans_finished: Total number of bucket scans finished since server start.

minio_node_scanner_bucket_scans_started: Total number of bucket scans started since server start.

minio_node_scanner_directories_scanned: Total number of directories scanned since server start.

minio_node_scanner_objects_scanned: Total number of unique objects scanned since server start.

minio_node_scanner_versions_scanned: Total number of object versions scanned since server start.

minio_node_syscall_read_total: Total number of read SysCalls to the kernel. /proc/[pid]/io syscr

minio_node_syscall_write_total: Total number of write SysCalls to the kernel. /proc/[pid]/io syscw

minio_usage_last_activity_nano_seconds: Time elapsed since last scan activity. This is set to 0 until first scan cycle.

S3 Metrics

minio_bucket_traffic_sent_bytes: Total number of bytes of S3 traffic sent per bucket. You can identify the bucket using the { bucket="STRING" } label.

minio_bucket_traffic_received_bytes: Total number of bytes of S3 traffic received per bucket. You can identify the bucket using the { bucket="STRING" } label.

minio_s3_requests_incoming_total: Volatile number of total incoming S3 requests.

minio_s3_requests_canceled_total: Total number S3 requests that were canceled from the client while processing.

minio_s3_requests_inflight_total: Total number of S3 requests currently in flight.

minio_s3_requests_total: Total number of S3 requests.

minio_s3_requests_rejected_auth_total: Total number S3 requests rejected for auth failure.

minio_s3_requests_rejected_header_total: Total number S3 requests rejected for invalid header.

minio_s3_requests_rejected_invalid_total: Total number S3 invalid requests.

minio_s3_requests_rejected_timestamp_total: Total number S3 requests rejected for invalid timestamp.

minio_s3_requests_waiting_total: Number of S3 requests in the waiting queue.

minio_s3_time_ttfb_seconds_distribution: Distribution of the time to first byte across API calls.

minio_s3_traffic_received_bytes: Total number of S3 bytes received.

minio_s3_traffic_sent_bytes: Total number of S3 bytes sent.

minio_s3_requests_errors_total: Total number of S3 requests with 4xx and 5xx errors.

minio_s3_requests_4xx_errors_total: Total number of S3 requests with 4xx errors.

minio_s3_requests_5xx_errors_total: Total number of S3 requests with 5xx errors.

IAM Metrics

minio_node_iam_last_sync_duration_millis: Last successful IAM data sync duration in milliseconds.

minio_node_iam_since_last_sync_millis

Time (in milliseconds) since last successful IAM data sync.

This value starts at zero and only increments after the the first sync after server start.

minio_node_iam_sync_failures: Number of failed IAM data syncs since server start.

minio_node_iam_sync_successes: Number of successful IAM data syncs since server start.

Internal Metrics

minio_inter_node_traffic_received_bytes: Total number of bytes received from other peer nodes.

minio_inter_node_traffic_sent_bytes: Total number of bytes sent to the other peer nodes.

minio_inter_node_traffic_dial_avg_time: Average time of internodes TCP dial calls.

minio_inter_node_traffic_dial_errors: Total number of internode TCP dial timeouts and errors.

minio_inter_node_traffic_errors_total: Total number of failed internode calls.

minio_node_file_descriptor_limit_total: Limit on total number of open file descriptors for the MinIO Server process.

minio_node_file_descriptor_open_total: Total number of open file descriptors by the MinIO Server process.

minio_node_io_rchar_bytes: Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar

minio_node_io_read_bytes: Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes

minio_node_io_wchar_bytes: Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar

minio_node_io_write_bytes: Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes

Key Management System (KMS) Metrics

minio_cluster_kms_online: Reports whether the KMS is online (1) or offline (0).

minio_cluster_kms_request_error: Number of KMS requests that failed due to some error. (HTTP 4xx status code).

minio_cluster_kms_request_failure: Number of KMS requests that failed due to some internal failure. (HTTP 5xx status code).

minio_cluster_kms_request_success: Number of KMS requests that succeeded.

minio_cluster_kms_uptime: The time the KMS has been up and running in seconds.

Software and Process Metrics

minio_software_commit_info: Git commit hash for the MinIO release.

minio_software_version_info: MinIO Release tag for the server

minio_node_go_routine_total: Total number of go routines running.

minio_node_process_starttime_seconds: Start time for MinIO process per node, time in seconds since Unix epoch.

minio_node_process_uptime_seconds: Uptime for MinIO process per node in seconds.

minio_node_process_cpu_total_seconds: Total user and system CPU time spent in seconds.

minio_node_process_resident_memory_bytes: Resident memory size in bytes.