Documentation

Hardware Checklist

Use the following checklist when planning the hardware configuration for a production, distributed MinIO deployment.

Considerations

When selecting hardware for your MinIO implementation, take into account the following factors:

  • Expected amount of data in tebibytes to store at launch

  • Expected growth in size of data for at least the next two years

  • Number of objects by average object size

  • Average retention time of data in years

  • Number of sites to be deployed

  • Number of expected buckets

Production Hardware Recommendations

The following checklist follows MinIO’s Recommended Configuration for production deployments. The provided guidance is intended as a baseline and cannot replace MinIO SUBNET Performance Diagnostics, Architecture Reviews, and direct-to-engineering support.

MinIO, like any distributed system, benefits from selecting identical configurations for all nodes in a given server pool. Ensure a consistent selection of hardware (CPU, memory, motherboard, storage adapters) and software (operating system, kernel settings, system services) across pool nodes.

Deployments may exhibit unpredictable performance if nodes have varying hardware or software configurations. Workloads that benefit from storing aged data on lower-cost hardware should instead deploy a dedicated “warm” or “cold” MinIO deployment and transition data to that tier.

MinIO does not provide hosted services or hardware sales

See our Reference Hardware page for a curated selection of servers and storage components from our hardware partners.

Description

Minimum

Recommended

Dedicated Baremetal or Virtual Hosts (“hosts”).

4 dedicated hosts

8+ dedicated hosts

Dedicated locally-attached drives for each host.

4 drives per MinIO Server

8+ drives per MinIO Server

High speed network infrastructure.

25GbE

100GbE

Server-grade CPUs with support for modern SIMD instructions (AVX-512), such as Intel® Xeon® Scalable or better.

8 CPU/socket or vCPU per host

16+ CPU/socket or vCPU per host

Available memory to meet or exceed per-server usage by a reasonable buffer.

32GB of available memory per host

128GB+ of available memory per host

Important

The following areas have the greatest impact on MinIO performance, listed in order of importance:

Network Infrastructure

Insufficient or limited throughput constrains performance

Storage Controller

Old firmware, limited throughput, or failing hardware constrains performance and affects reliability

Storage (Drive)

Old firmware, or slow/aging/failing hardware constrains performance and affects reliability

Prioritize securing the necessary components for each of these areas before focusing on other hardware resources, such as compute-related constraints.

The minimum recommendations above reflect MinIO’s experience with assisting enterprise customers in deploying on a variety of IT infrastructures while maintaining the desired SLA/SLO. While MinIO may run on less than the minimum recommended topology, any potential cost savings come at the risk of decreased reliability, performance, or overall functionality.

Networking

MinIO recommends high speed networking to support the maximum possible throughput of the attached storage (aggregated drives, storage controllers, and PCIe busses). The following table provides a general guideline for the maximum storage throughput supported by a given physical or virtual network interface. This table assumes all network infrastructure components, such as routers, switches, and physical cabling, also supports the NIC bandwidth.

NIC Bandwidth (Gbps)

Estimated Aggregated Storage Throughput (GBps)

10Gbps

1.25GBps

25Gbps

3.125GBps

50Gbps

6.25GBps

100Gbps

12.5GBps

Networking has the greatest impact on MinIO performance, where low per-host bandwidth artificially constrains the potential performance of the storage. The following examples of network throughput constraints assume spinning disks with ~100MB/S sustained I/O

  • 1GbE network link can support up to 125MB/s, or one spinning disk

  • 10GbE network can support approximately 1.25GB/s, potentially supporting 10-12 spinning disks

  • 25GbE network can support approximately 3.125GB/s, potentially supporting ~30 spinning disks

Memory

Memory primarily constrains the number of concurrent simultaneous connections per node.

You can calculate the maximum number of concurrent requests per node with this formula:

\(totalRam / ramPerRequest\)

To calculate the amount of RAM used for each request, use this formula:

\(((2MiB + 128KiB) * driveCount) + (2 * 10MiB) + (2 * 1 MiB)\)

10MiB is the default erasure block size v1. 1 MiB is the default erasure block size v2.

The following table lists the maximum concurrent requests on a node based on the number of host drives and the free system RAM:

Number of Drives

32 GiB of RAM

64 GiB of RAM

128 GiB of RAM

256 GiB of RAM

512 GiB of RAM

4 Drives

1,074

2,149

4,297

8,595

17,190

8 Drives

840

1,680

3,361

6,722

13,443

16 Drives

585

1,170

2.341

4,681

9,362

The following table provides general guidelines for allocating memory for use by MinIO based on the total amount of local storage on the node:

Total Host Storage

Recommended Host Memory

Up to 1 Tebibyte (Ti)

8GiB

Up to 10 Tebibyte (Ti)

16GiB

Up to 100 Tebibyte (Ti)

32GiB

Up to 1 Pebibyte (Pi)

64GiB

More than 1 Pebibyte (Pi)

128GiB

Important

Starting with RELEASE.2024-01-28T22-35-53Z, MinIO preallocates 2GiB of memory per node in distributed setups and 1GiB of memory for a single-node setup.

Storage

Exclusive access to drives

MinIO requires exclusive access to the drives or volumes provided for object storage. No other processes, software, scripts, or persons should perform any actions directly on the drives or volumes provided to MinIO or the objects or files MinIO places on them.

Unless directed by MinIO Engineering, do not use scripts or tools to directly modify, delete, or move any of the data shards, parity shards, or metadata files on the provided drives, including from one drive or node to another. Such operations are very likely to result in widespread corruption and data loss beyond MinIO’s ability to heal.

Use Direct-Attached “Local” Storage (DAS)

DAS, such as locally-attached JBOD (Just a Bunch of Disks) arrays, provide significant performance and consistency advantages over networked (NAS, SAN, NFS) storage.

Network File System Volumes Break Consistency Guarantees

MinIO’s strict read-after-write and list-after-write consistency model requires local drive filesystems. MinIO cannot provide consistency guarantees if the underlying storage volumes are NFS or a similar network-attached storage volume.

Use XFS-Formatted Drives with Labels

Format drives as XFS and present them to MinIO as a JBOD array with no RAID or other pooling configurations. Using any other type of backing storage (SAN/NAS, ext4, RAID, LVM) typically results in a reduction in performance, reliability, predictability, and consistency.

When formatting XFS drives, apply a unique label per drive. For example, the following command formats four drives as XFS and applies a corresponding drive label.

mkfs.xfs /dev/sdb -L MINIODRIVE1
mkfs.xfs /dev/sdc -L MINIODRIVE2
mkfs.xfs /dev/sdd -L MINIODRIVE3
mkfs.xfs /dev/sde -L MINIODRIVE4

Mount Drives using /etc/fstab

MinIO requires that drives maintain their ordering at the mounted position across restarts. MinIO does not support arbitrary migration of a drive with existing MinIO data to a new mount position, whether intentional or as the result of OS-level behavior.

You must use /etc/fstab or a similar mount control system to mount drives at a consistent path. For example:

$ nano /etc/fstab

# <file system>        <mount point>    <type>  <options>         <dump>  <pass>
LABEL=MINIODRIVE1      /mnt/drive-1     xfs     defaults,noatime  0       2
LABEL=MINIODRIVE2      /mnt/drive-2     xfs     defaults,noatime  0       2
LABEL=MINIODRIVE3      /mnt/drive-3     xfs     defaults,noatime  0       2
LABEL=MINIODRIVE4      /mnt/drive-4     xfs     defaults,noatime  0       2

You can use mount -a to mount those drives at those paths during initial setup. The Operating System should otherwise mount these drives as part of the node startup process.

MinIO strongly recommends using label-based mounting rules over UUID-based rules. Label-based rules allow swapping an unhealthy or non-working drive with a replacement that has matching format and label. UUID-based rules require editing the /etc/fstab file to replace mappings with the new drive UUID.

Note

Cloud environment instances which depend on mounted external storage may encounter boot failure if one or more of the remote file mounts return errors or failure. For example, an AWS ECS instance with mounted persistent EBS volumes may not boot with the standard /etc/fstab configuration if one or more EBS volumes fail to mount.

You can set the nofail option to silence error reporting at boot and allow the instance to boot with one or more mount issues.

You should not use this option on systems with locally attached disks, as silencing drive errors prevents both MinIO and the OS from responding to those errors in a normal fashion.

Disable XFS Retry On Error

MinIO strongly recommends disabling retry-on-error behavior using the max_retries configuration for the following error classes:

  • EIO Error when reading or writing

  • ENOSPC Error no space left on device

  • default All other errors

The default max_retries setting typically directs the filesystem to retry-on-error indefinitely instead of propagating the error. MinIO can handle XFS errors appropriately, such that the retry-on-error behavior introduces at most unnecessary latency or performance degradation.

The following script iterates through all drives at the specified mount path and sets the XFS max_retries setting to 0 or “fail immediately on error” for the recommended error classes. The script ignores any drives not mounted, either manually or through /etc/fstab. Modify the /mnt/drive line to match the pattern used for your MinIO drives.

#!/bin/bash

for i in $(df -h | grep /mnt/drive | awk '{ print $1 }'); do
      mountPath="$(df -h | grep $i | awk '{ print $6 }')"
      deviceName="$(basename $i)"
      echo "Modifying xfs max_retries and retry_timeout_seconds for drive $i mounted at $mountPath"
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/EIO/max_retries
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/ENOSPC/max_retries
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/default/max_retries
done
exit 0

You must run this script on all MinIO nodes and configure the script to re-run on reboot, as Linux Operating Systems do not typically persist these changes. You can use a cron job with the @reboot timing to run the above script whenever the node restarts and ensure all drives have retry-on-error disabled. Use crontab -e to create the following job, modifying the script path to match that on each node:

@reboot /opt/minio/xfs-retry-settings.sh

Use Consistent Drive Type and Capacity

Ensure a consistent drive type (NVMe, SSD, HDD) for the underlying storage in a MinIO deployment. MinIO does not distinguish between storage types and does not support configuring “hot” or “warm” drives within a single deployment. Mixing drive types typically results in performance degradation, as the slowest drives in the deployment become a bottleneck regardless of the capabilities of the faster drives.

Use the same capacity and type of drive across all nodes in each MinIO server pool. MinIO limits the maximum usable size per drive to the smallest size in the deployment. For example, if a deployment has 15 10TB drives and 1 1TB drive, MinIO limits the per-drive capacity to 1TB.