Expand a Distributed MinIO Deployment
MinIO supports expanding an existing distributed deployment by adding a new Server Pool. Each Pool expands the total available storage capacity of the cluster.
Expansion does not provide Business Continuity/Disaster Recovery (BC/DR)-grade protections. While each pool is an independent set of servers with distinct erasure sets for availability, the complete loss of one pool results in MinIO stopping I/O for all pools in the deployment. Similarly, an erasure set which loses quorum in one pool represents data loss of objects stored in that set, regardless of the number of other erasure sets or pools.
To provide BC-DR grade failover and recovery support for your single or multi-pool MinIO deployments, use site replication.
The procedure on this page expands an existing distributed MinIO deployment with an additional server pool.
Prerequisites
Networking and Firewalls
Each node should have full bidirectional network access to every other node in
the deployment. For containerized or orchestrated infrastructures, this may
require specific configuration of networking and routing components such as
ingress or load balancers. Certain operating systems may also require setting
firewall rules. For example, the following command explicitly opens the default
MinIO server API port 9000
on servers using firewalld
:
firewall-cmd --permanent --zone=public --add-port=9000/tcp
firewall-cmd --reload
All MinIO servers in the deployment must use the same listen port.
If you set a static MinIO Console port (e.g. :9001
)
you must also grant access to that port to ensure connectivity from external
clients.
MinIO strongly recomends using a load balancer to manage connectivity to the cluster. The Load Balancer should use a “Least Connections” algorithm for routing requests to the MinIO deployment, since any MinIO node in the deployment can receive, route, or process client requests.
The following load balancers are known to work well with MinIO:
Configuring firewalls or load balancers to support MinIO is out of scope for this procedure. The Configure NGINX Proxy for MinIO Server reference provides a baseline configuration for using NGINX as a reverse proxy with basic load balancing configured.
Sequential Hostnames
MinIO requires using expansion notation {x...y}
to denote a sequential
series of MinIO hosts when creating a server pool. MinIO therefore requires
using sequentially-numbered hostnames to represent each
minio server
process in the pool.
Create the necessary DNS hostname mappings prior to starting this procedure. For example, the following hostnames would support a 4-node distributed server pool:
minio5.example.com
minio6.example.com
minio7.example.com
minio8.example.com
You can specify the entire range of hostnames using the expansion notation
minio{5...8}.example.com
.
Configuring DNS to support MinIO is out of scope for this procedure.
Local JBOD Storage with Sequential Mounts
MinIO strongly recommends direct-attached JBOD arrays with XFS-formatted disks for best performance.
Direct-Attached Storage (DAS) has significant performance and consistency advantages over networked storage (NAS, SAN, NFS).
Deployments using non-XFS filesystems (ext4, btrfs, zfs) tend to have lower performance while exhibiting unexpected or undesired behavior.
RAID or similar technologies do not provide additional resilience or availability benefits when used with distributed MinIO deployments, and typically reduce system performance.
Ensure all nodes in the server pool use the same type (NVMe, SSD, or HDD) of
drive with identical capacity (e.g. N
TB) . MinIO does not distinguish drive
types and does not benefit from mixed storage types. Additionally. MinIO limits
the size used per drive to the smallest drive in the deployment. For example, if
the deployment has 15 10TB drives and 1 1TB drive, MinIO limits the per-drive
capacity to 1TB.
MinIO requires using expansion notation {x...y}
to denote a sequential
series of drives when creating the new server pool, where all nodes in the
server pool have an identical set of mounted drives. MinIO also
requires that the ordering of physical drives remain constant across restarts,
such that a given mount point always points to the same formatted drive. MinIO
therefore strongly recommends using /etc/fstab
or a similar file-based
mount configuration to ensure that drive ordering cannot change after a reboot.
For example:
$ mkfs.xfs /dev/sdb -L DISK1
$ mkfs.xfs /dev/sdc -L DISK2
$ mkfs.xfs /dev/sdd -L DISK3
$ mkfs.xfs /dev/sde -L DISK4
$ nano /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
LABEL=DISK1 /mnt/disk1 xfs defaults,noatime 0 2
LABEL=DISK2 /mnt/disk2 xfs defaults,noatime 0 2
LABEL=DISK3 /mnt/disk3 xfs defaults,noatime 0 2
LABEL=DISK4 /mnt/disk4 xfs defaults,noatime 0 2
You can then specify the entire range of drives using the expansion notation
/mnt/disk{1...4}
. If you want to use a specific subfolder on each drive,
specify it as /mnt/disk{1...4}/minio
.
MinIO does not support arbitrary migration of a drive with existing MinIO data to a new mount position, whether intentional or as the result of OS-level behavior.
Note
Cloud environment instances which depend on mounted external storage may encounter boot failure if one or more of the remote file mounts return errors or failure.
For example, an AWS ECS instances with mounted persistent EBS volumes may fail to boot with the standard /etc/fstab
configuration if one or more EBS volumes fail to mount.
You can set the nofail
option to silence error reporting at boot and allow the instance to boot with one or more mount issues.
You should not use this option on systems which have locally attached disks, as silencing drive errors prevents both MinIO and the OS from responding to those errors in a normal fashion.
Network File System Volumes Break Consistency Guarantees
MinIO’s strict read-after-write and list-after-write consistency
model requires local drive filesystems (xfs
, ext4
, etc.).
MinIO cannot provide consistency guarantees if the underlying storage volumes are NFS or a similar network-attached storage volume.
For deployments that require using network-attached storage, use NFSv4 for best results.
Minimum Drives for Erasure Code Parity
MinIO requires each pool satisfy the deployment erasure code settings. Specifically the new pool topology must
support a minimum of 2 x EC:N
drives per
erasure set, where EC:N
is the
Standard parity storage class of the
deployment. This requirement ensures the new server pool can satisfy the
expected SLA of the deployment.
You can use the
MinIO Erasure Code Calculator to check the
Erasure Code Stripe Size (K+M) of your new pool. If the highest
listed value is at least 2 x EC:N
, the pool supports the deployment’s
erasure parity settings.
Considerations
Writing Files
MinIO does not automatically rebalance objects across the new server pools. Instead, MinIO performs new write operations to the pool with the most free storage weighted by the amount of free space on the pool divided by the free space across all available pools.
The formula to determine the probability of a write operation on a particular pool is
\(FreeSpaceOnPoolA / FreeSpaceOnAllPools\)
Consider a situation where a group of two pools has a total of 10 TiB of free space distributed as:
Pool A has 3 TiB of free space
Pool B has 2 TiB of free space
Pool C has 5 TiB of free space
MinIO calculates the probability of a write operation to each of the pools as:
Pool A: 30% chance (\(3TiB / 10TiB\))
Pool B: 20% chance (\(2TiB / 10TiB\))
Pool C: 50% chance (\(5TiB / 10TiB\))
In addition to the free space calculation, if a write option (with parity) would bring a drive usage above 99% or a known free inode count below 1000, MinIO does not write to the pool.
If desired, you can manually initiate a rebalance procedure with mc admin rebalance
.
For more about how rebalancing works, see managing objects across a deployment.
Likewise, MinIO does not write to pools in a decommissioning process.
Homogeneous Node Configurations
MinIO strongly recommends selecting substantially similar hardware configurations for all nodes in the new server pool. Ensure the hardware (CPU, memory, motherboard, storage adapters) and software (operating system, kernel settings, system services) is consistent across all nodes in the pool.
The new pool may exhibit unpredictable performance if nodes have heterogeneous hardware or software configurations. Workloads that benefit from storing aged data on lower-cost hardware should instead deploy a dedicated “warm” or “cold” MinIO deployment and transition data to that tier.
The new server pool does not need to be substantially similar in hardware and software configuration to any existing server pool, though this may allow for simplified cluster management and more predictable performance across pools.
See Production Hardware Recommendations for more guidance on selecting hardware for MinIO deployments.
Expansion is Non-Disruptive
Adding a new server pool requires restarting all MinIO nodes in the deployment at around same time.
MinIO strongly recommends restarting all nodes simultaneously. MinIO operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.
Do not perform “rolling” (e.g. one node at a time) restarts.
Capacity-Based Planning
MinIO generally recommends planning capacity such that server pool expansion is only required after 2+ years of deployment uptime.
For example, consider an application suite that is estimated to produce 10TB of data per year. The current deployment is running low on free storage and therefore requires expansion to meet the ongoing storage demands of the application. The new server pool should provide at minimum
10TB + 10TB + 10TB = 30TB
MinIO recommends adding buffer storage to account for potential growth in stored data (e.g. 40TB of total usable storage). The total planned usable storage in the deployment would therefore be ~80TB. As a rule-of-thumb, more capacity initially is preferred over frequent just-in-time expansion to meet capacity requirements.
Since MinIO erasure coding requires some storage for parity, the total raw storage must exceed the planned usable capacity. Consider using the MinIO Erasure Code Calculator for guidance in planning capacity around specific erasure code settings.
Recommended Operating Systems
This tutorial assumes all hosts running MinIO use a recommended Linux operating system such as RHEL8+ or Ubuntu 18.04+.
For other operating systems such as Windows or OSX, visit https://min.io/download and select the tab associated to your operating system. Follow the displayed instructions to install the MinIO server binary on each node. Defer to the OS best practices for starting MinIO as a service (e.g. not attached to the terminal/shell session).
Support for running MinIO in distributed mode on Windows hosts is experimental. Contact MinIO at hello@min.io if your infrastructure requires deployment onto Windows hosts.
Expand a Distributed MinIO Deployment
The following procedure adds a Server Pool to an existing MinIO deployment. Each Pool expands the total available storage capacity of the cluster while maintaining the overall availability of the cluster.
All commands provided below use example values. Replace these values with those appropriate for your deployment.
Review the Prerequisites before starting this procedure.
Complete any planned hardware expansion prior to decommissioning older hardware pools.
1) Install the MinIO Binary on Each Node in the New Server Pool
The following tabs provide examples of installing MinIO onto 64-bit Linux operating systems using RPM, DEB, or binary.
The RPM and DEB packages automatically install MinIO to the necessary system paths and create a minio
service for systemctl
.
MinIO strongly recommends using the RPM or DEB installation routes.
To update deployments managed using systemctl
, see Update systemctl-Managed MinIO Deployments.
amd64 (Intel or AMD 64-bit processors)
Use one of the following options to download the MinIO server installation file for a machine running Linux on an Intel or AMD 64-bit processor.
Use the following commands to download the latest stable MinIO RPM and install it.
wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio-20231120224007.0.0.x86_64.rpm -O minio.rpm
sudo dnf install minio.rpm
Use the following commands to download the latest stable MinIO DEB and install it:
wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20231120224007.0.0_amd64.deb -O minio.deb
sudo dpkg -i minio.deb
Use the following commands to download the latest stable MinIO binary and
install it to the system $PATH
:
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/
arm64 (Apple M1/M2 or other ARM 64-bit processors)
Use one of the following options to download the MinIO server installation file for a machine running Linux on an ARM 64-bit processor, such as the Apple M1 or M2.
Use the following commands to download the latest stable MinIO RPM and install it.
wget https://dl.min.io/server/minio/release/linux-arm64/archive/minio-20231120224007.0.0.aarch64.rpm -O minio.rpm
sudo dnf install minio.rpm
Use the following commands to download the latest stable MinIO DEB and install it:
wget https://dl.min.io/server/minio/release/linux-arm64/archive/minio_20231120224007.0.0_arm64.deb -O minio.deb
sudo dpkg -i minio.deb
Use the following commands to download the latest stable MinIO binary and
install it to the system $PATH
:
wget https://dl.min.io/server/minio/release/linux-arm64/minio
chmod +x minio
MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=password ./minio server /mnt/data --console-address ":9001"
Other Architectures
MinIO also supports additional architectures:
ppc64le
s390x
For instructions to download the binary, RPM, or DEB files for those architectures, see the MinIO download page.
2) Add TLS/SSL Certificates
MinIO enables Transport Layer Security (TLS) 1.2+
automatically upon detecting a valid x.509 certificate (.crt
) and
private key (.key
) in the MinIO ${HOME}/.minio/certs
directory.
For systemd
-managed deployments, use the $HOME
directory for the
user which runs the MinIO server process. The provided minio.service
file runs the process as minio-user
. The previous step includes instructions
for creating this user with a home directory /home/minio-user
.
Place TLS certificates into
/home/minio-user/.minio/certs
on each host.If any MinIO server or client uses certificates signed by an unknown Certificate Authority (self-signed or internal CA), you must place the CA certs in the
/home/minio-user/.minio/certs/CAs
on all MinIO hosts in the deployment. MinIO rejects invalid certificates (untrusted, expired, or malformed).
If the minio.service
file specifies a different user account, use the
$HOME
directory for that account. Alternatively, specify a custom
certificate directory using the minio server --certs-dir
commandline argument. Modify the MINIO_OPTS
variable in
/etc/defaults/minio
to set this option. The systemd
user which runs the
MinIO server process must have read and listing permissions for the specified
directory.
For more specific guidance on configuring MinIO for TLS, including multi-domain support via Server Name Indication (SNI), see Network Encryption (TLS). You can optionally skip this step to deploy without TLS enabled. MinIO strongly recommends against non-TLS deployments outside of early development.
3) Create the systemd
Service File
The .deb
or .rpm
packages install the following systemd service file to /usr/lib/systemd/system/minio.service
.
For binary installations, create this file manually on all MinIO hosts.
Note
systemd
checks the /etc/systemd/...
path before checking the /usr/lib/systemd/...
path and uses the first file it finds.
To avoid conflicting or unexpected configuration options, check that the file only exists at the /usr/lib/systemd/system/minio.service
path.
Refer to the man page for systemd.unit for details on the file path search order.
[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local
User=minio-user
Group=minio-user
ProtectProc=invisible
EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
# MinIO RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Specifies the maximum number of threads this process can create
TasksMax=infinity
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
# Built for ${project.name}-${project.version} (${project.name})
The minio.service
file runs as the minio-user
User and Group by default.
You can create the user and group using the groupadd
and useradd
commands. The following example creates the user, group, and sets permissions
to access the folder paths intended for use by MinIO. These commands typically
require root (sudo
) permissions.
groupadd -r minio-user
useradd -M -r -g minio-user minio-user
chown minio-user:minio-user /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4
The specified drive paths are provided as an example. Change them to match the path to those drives intended for use by MinIO.
Alternatively, change the User
and Group
values to another user and
group on the system host with the necessary access and permissions.
MinIO publishes additional startup script examples on github.com/minio/minio-service.
To update deployments managed using systemctl
, see Update systemctl-Managed MinIO Deployments.
4) Create the Service Environment File
Create an environment file at /etc/default/minio
. The MinIO
service uses this file as the source of all
environment variables used by
MinIO and the minio.service
file.
The following examples assumes that:
The deployment has a single server pool consisting of four MinIO server hosts with sequential hostnames.
minio1.example.com minio3.example.com minio2.example.com minio4.example.com
Each host has 4 locally attached drives with sequential mount points:
/mnt/disk1/minio /mnt/disk3/minio /mnt/disk2/minio /mnt/disk4/minio
The new server pool consists of eight new MinIO hosts with sequential hostnames:
minio5.example.com minio9.example.com minio6.example.com minio10.example.com minio7.example.com minio11.example.com minio8.example.com minio12.example.com
All hosts have eight locally-attached drives with sequential mount-points:
/mnt/disk1/minio /mnt/disk5/minio /mnt/disk2/minio /mnt/disk6/minio /mnt/disk3/minio /mnt/disk7/minio /mnt/disk4/minio /mnt/disk8/minio
The deployment has a load balancer running at
https://minio.example.net
that manages connections across all MinIO hosts. The load balancer should not be routing requests to the new hosts at this step, but should have the necessary configuration updates planned.
Modify the example to reflect your deployment topology:
# Set the hosts and volumes MinIO uses at startup
# The command uses MinIO expansion notation {x...y} to denote a
# sequential series.
#
# The following example starts the MinIO server with two server pools.
#
# The space delimiter indicates a seperate server pool
#
# The second set of hostnames and volumes is the newly added pool.
# The pool has sufficient stripe size to meet the existing erasure code
# parity of the deployment (2 x EC:4)
#
# The command includes the port on which the MinIO servers listen for each
# server pool.
MINIO_VOLUMES="https://minio{1...4}.example.net:9000/mnt/disk{1...4}/minio https://minio{5...12}.example.net:9000/mnt/disk{1...8}/minio"
# Set all MinIO server options
#
# The following explicitly sets the MinIO Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.
MINIO_OPTS="--console-address :9001"
# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.
MINIO_ROOT_USER=minioadmin
# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.
MINIO_ROOT_PASSWORD=minio-secret-key-CHANGE-ME
# Set to the URL of the load balancer for the MinIO deployment
# This value *must* match across all MinIO servers. If you do
# not have a load balancer, set this value to to any *one* of the
# MinIO hosts in the deployment as a temporary measure.
MINIO_SERVER_URL="https://minio.example.net:9000"
You may specify other environment variables or server commandline options as required by your deployment. All MinIO nodes in the deployment should include the same environment variables with the matching values.
5) Restart the MinIO Deployment with Expanded Configuration
Issue the following commands on each node simultaneously in the deployment to restart the MinIO service:
sudo systemctl restart minio.service
Use the following commands to confirm the service is online and functional:
sudo systemctl status minio.service
journalctl -f -u minio.service
MinIO may log an increased number of non-critical warnings while the server processes connect and synchronize. These warnings are typically transient and should resolve as the deployment comes online.
MinIO strongly recommends restarting all nodes simultaneously. MinIO operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.
Do not perform “rolling” (e.g. one node at a time) restarts.
6) Next Steps
Update any load balancers, reverse proxies, or other network control planes to route client requests to the new hosts in the MinIO distributed deployment. While MinIO automatically manages routing internally, having the control planes handle initial connection management may reduce network hops and improve efficiency.
Review the MinIO Console to confirm the updated cluster topology and monitor performance.