Decommission Server Pools

Back Up Cluster Settings First

Use the mc admin cluster bucket export and mc admin cluster iam export commands to take a snapshot of the bucket metadata and IAM configurations respectively prior to starting decommissioning. You can use these snapshots to restore bucket/IAM settings to recover from user or process errors as necessary.

Networking and Firewalls

Each node should have full bidirectional network access to every other node in the deployment. For containerized or orchestrated infrastructures, this may require specific configuration of networking and routing components such as ingress or load balancers. Certain operating systems may also require setting firewall rules. For example, the following command explicitly opens the default MinIO server API port 9000 on servers using firewalld:

firewall-cmd --permanent --zone=public --add-port=9000/tcp
firewall-cmd --reload

If you set a static MinIO Console port (e.g. :9001) you must also grant access to that port to ensure connectivity from external clients.

MinIO strongly recomends using a load balancer to manage connectivity to the cluster. The Load Balancer should use a “Least Connections” algorithm for routing requests to the MinIO deployment, since any MinIO node in the deployment can receive, route, or process client requests.

The following load balancers are known to work well with MinIO:

• NGINX

• HAProxy

Configuring firewalls or load balancers to support MinIO is out of scope for this procedure.

Deployment Must Have Sufficient Storage

The decommissioning process migrates objects from the target pool to other pools in the deployment. The total available storage on the deployment must exceed the total storage of the decommissioned pool.

Use the Erasure Code Calculator to determine the usable storage capacity. Then reduce that by the size of the objects already on the deployment.

For example, consider a deployment with the following distribution of used and free storage:

Decommissioning Pool 1 requires distributing the 100TB of used storage across the remaining pools. Pool 2 and Pool 3 each have 100TB of unused storage space and can safely absorb the data stored on Pool 1.

However, if Pool 1 were full (e.g. 200TB of used space), decommissioning would completely fill the remaining pools and potentially prevent any further write operations.

Replacing a Server Pool

For hardware upgrade cycles where you replace old pool hardware with a new pool, you should add the new pool through expansion before starting the decommissioning of the old pool. Adding the new pool first allows the decommission process to transfer objects in a balanced way across all available pools, both existing and new.

Complete any planned hardware expansion prior to decommissioning older hardware pools.

Decommissioning requires that a cluster’s topology remain stable throughout the pool draining process. Do not attempt to perform expansion and decommission changes in a single step.

Decommissioning is Resumable

MinIO resumes decommissioning if interrupted by transient issues such as deployment restarts or network failures.

For manually cancelled or failed decommissioning attempts, MinIO resumes only after you manually re-initiate the decommissioning operation.

The pool remains in the decommissioning state regardless of the interruption. A pool can never return to active status after decommissioning begins.

Decommissioning is Non-Disruptive

Removing a decommissioned server pool requires restarting all MinIO nodes in the deployment at around the same time.

MinIO strongly recommends restarting all MinIO Server processes in a deployment simultaneously. MinIO operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.

Do not perform “rolling” (e.g. one node at a time) restarts.

Decommissioning Ignores Expired Objects and Trailing DeleteMarker

Starting with RELEASE.2023-05-27T05-56-19Z, decommissioning ignores objects where the only remaining version is a DeleteMarker. This avoids creating empty metadata on the remaining server pool(s) for objects that are effectively fully deleted.

Starting with RELEASE.2023-06-23T20-26-00Z, decommissioning also ignores object versions which have expired based on the configured lifecycle rules for the parent bucket. Starting with RELEASE.2023-06-29T05-12-28Z, you can monitor ignored delete markers and expired objects during the decommission process with mc admin trace --call decommission.

Once the decommissioning process completes, you can safely shut down that pool. Since the only remaining data was scheduled for deletion or was only a DeleteMarker, you can safely clear or destroy those drives as per your internal procedures.

Final Listing Check

At the end of the decommission process, MinIO checks for a list of items on the pool. If the list returns empty, MinIO marks the decommission as successfully completed. If any objects return, MinIO returns an error that the decommission process failed.

If the decommission fails, customers should open a MinIO SUBNET issue for further assistance before retrying the decommission. Community users without a SUBNET subscription can retry the decommission process or seek additional support through the MinIO Community Slack. MinIO provides Community Support at best-effort only and provides no SLA around responsiveness.

Decommissioning a Server with Tiering Enabled

For deployments with tiering enabled and active, decommissioning moves the object references to a new active pool. Applications can continue issuing GET requests against those objects where MinIO handles transparently retrieving them from the remote tier.

In older MinIO versions, tiering configurations prevent decommissioning.

1) Review the MinIO Deployment Topology

The mc admin decommission command returns a list of all pools in the MinIO deployment:

mc admin decommission status myminio

The command returns output similar to the following:

┌─────┬────────────────────────────────────────────────────────────────┬──────────────────────────────────┬────────┐
│ ID  │ Pools                                                          │ Capacity                         │ Status │
│ 1st │ https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio │  10 TiB (used) / 10  TiB (total) │ Active │
│ 2nd │ https://minio-{05...08}.example.com:9000/mnt/disk{1...4}/minio │  60 TiB (used) / 100 TiB (total) │ Active │
│ 3rd │ https://minio-{09...12}.example.com:9000/mnt/disk{1...4}/minio │  40 TiB (used) / 100 TiB (total) │ Active │
└─────┴────────────────────────────────────────────────────────────────┴──────────────────────────────────┴────────┘

The example deployment above has three pools. Each pool has four servers with four drives each.

Identify the target pool for decommissioning and review the current capacity. The remaining pools in the deployment must have sufficient total capacity to migrate all object stored in the decommissioned pool.

In the example above, the deployment has 210TiB total storage with 110TiB used. The first pool (minio-{01...04}) is the decommissioning target, as it was provisioned when the MinIO deployment was created and is completely full. The remaining newer pools can absorb all objects stored on the first pool without significantly impacting total available storage.

2) Start the Decommissioning Process

Use the mc admin decommission start command to begin decommissioning the target pool. Specify the alias of the deployment and the full description of the pool to decommission, including all hosts, disks, and file paths.

mc admin decommission start myminio/ https://minio-{01...04}.example.net:9000/mnt/disk{1...4}/minio

The example command begins decommissioning the matching server pool on the myminio deployment.

During the decommissioning process, MinIO continues routing read operations (GET, LIST, HEAD) to the pool for those objects not yet migrated. MinIO routes all new write operations (PUT) to the remaining pools in the deployment.

Load balancers, reverse proxy, or other network control components which manage connections to the deployment do not need to modify their configurations at this time.

3) Monitor the Decommissioning Process

Use the mc admin decommission status command to monitor the decommissioning process.

mc admin decommission status myminio

The command returns output similar to the following:

┌─────┬────────────────────────────────────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID  │ Pools                                                          │ Capacity                         │ Status   │
│ 1st │ https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio │  10 TiB (used) / 10  TiB (total) │ Draining │
│ 2nd │ https://minio-{05...08}.example.com:9000/mnt/disk{1...4}/minio │  60 TiB (used) / 100 TiB (total) │ Active   │
│ 3rd │ https://minio-{09...12}.example.com:9000/mnt/disk{1...4}/minio │  40 TiB (used) / 100 TiB (total) │ Active   │
└─────┴────────────────────────────────────────────────────────────────┴──────────────────────────────────┴──────────┘

You can retrieve more detailed information by specifying the description of the server pool to the command:

mc admin decommission status myminio https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio

The command returns output similar to the following:

Decommissioning rate at 100MiB/sec [1TiB/10TiB]
Started: 30 minutes ago

mc admin decommission status marks the Status as Complete once decommissioning is completed. You can move on to the next step once decommissioning is completed.

If Status reads as failed, you can re-run the mc admin decommission start command to resume the process. For persistent failures, use mc admin logs or review the systemd logs (e.g. journalctl -u minio) to identify more specific errors.

4) Remove the Decommissioned Pool from the Deployment Configuration

As each pool completes decommissioning, you can safely remove it from the deployment configuration. Modify the startup command for each remaining MinIO server in the deployment and remove the decommissioned pool.

The .deb or .rpm packages install a systemd service file to /lib/systemd/system/minio.service. For binary installations, this procedure assumes the file was created manually as per the Deploy MinIO: Multi-Node Multi-Drive procedure.

The minio.service file uses an environment file located at /etc/default/minio for sourcing configuration settings, including the startup. Specifically, the MINIO_VOLUMES variable sets the startup command:

cat /etc/default/minio | grep "MINIO_VOLUMES"

The command returns output similar to the following:

MINIO_VOLUMES="https://minio-{1...4}.example.net:9000/mnt/disk{1...4}/minio https://minio-{5...8}.example.net:9000/mnt/disk{1...4}/minio https://minio-{9...12}.example.net:9000/mnt/disk{1...4}/minio"

Edit the environment file and remove the decommissioned pool from the MINIO_VOLUMES value.

5) Update Network Control Plane

Update any load balancers, reverse proxies, or other network control planes to remove the decommissioned server pool from the connection configuration for the MinIO deployment.

Specific instructions for configuring network control plane components is out of scope for this procedure.

6) Restart the MinIO Deployment

Issue the following commands on each node simultaneously in the deployment to restart the MinIO service:

sudo systemctl restart minio.service

Use the following commands to confirm the service is online and functional:

sudo systemctl status minio.service
journalctl -f -u minio.service

MinIO may log an increased number of non-critical warnings while the server processes connect and synchronize. These warnings are typically transient and should resolve as the deployment comes online.

MinIO strongly recommends restarting all MinIO Server processes in a deployment simultaneously. MinIO operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.

Do not perform “rolling” (e.g. one node at a time) restarts.

Once the deployment is online, use mc admin info to confirm the uptime of all remaining servers in the deployment.

1) Review the MinIO Deployment Topology

The mc admin decommission command returns a list of all pools in the MinIO deployment:

mc admin decommission status myminio

The command returns output similar to the following:

┌─────┬────────────────────────────────────────────────────────────────┬──────────────────────────────────┬────────┐
│ ID  │ Pools                                                          │ Capacity                         │ Status │
│ 1st │ https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio │  10 TiB (used) / 10  TiB (total) │ Active │
│ 2nd │ https://minio-{05...08}.example.com:9000/mnt/disk{1...4}/minio │  95 TiB (used) / 100 TiB (total) │ Active │
│ 3rd │ https://minio-{09...12}.example.com:9000/mnt/disk{1...4}/minio │  40 TiB (used) / 500 TiB (total) │ Active │
│ 4th │ https://minio-{13...16}.example.com:9000/mnt/disk{1...4}/minio │  0  TiB (used) / 500 TiB (total) │ Active │
└─────┴────────────────────────────────────────────────────────────────┴──────────────────────────────────┴────────┘

The example deployment above has three pools. Each pool has four servers with four drives each.

Identify the target pool for decommissioning and review the current capacity. The remaining pools in the deployment must have sufficient total capacity to migrate all object stored in the decommissioned pool.

In the example above, the deployment has 1110TiB total storage with 145TiB used.

• The first pool (minio-{01...04}) is the first decommissioning target, as it was provisioned when the MinIO deployment was created and is completely full.

• The second pool (minio-{05...08}) is the second decommissioning target, as it was also provisioned when the MinIO deployment was created and is nearly full.

• The fourth pool (minio-{13...16}) is a newly added pool with new hardware from a completed server expansion.

The third and fourth pools can absorb all objects stored on the first pool without significantly impacting total available storage.

2) Start the Decommissioning Process

Use the mc admin decommission start command to begin decommissioning the target pool. Specify the alias of the deployment and a comma-separated list of the full description of each pool to decommission, including all hosts, disks, and file paths.

mc admin decommission start myminio/ https://minio-{01...04}.example.net:9000/mnt/disk{1...4}/minio,https://minio-{05...08}.example.net:9000/mnt/disk{1...4}/minio

The example command begins decommissioning the two listed matching server pools on the myminio deployment.

During the decommissioning process, MinIO continues routing read operations (GET, LIST, HEAD) operations to the pools for those objects not yet migrated. MinIO routes all new write operations (PUT) to the remaining pools in the deployment not scheduled for decommissioning.

Draining of decommissioned pools happens one pool at a time, completing the decommission of each pool in sequence. Draining does _not_ happen concurrently for all decommissioning pools.

Load balancers, reverse proxy, or other network control components which manage connections to the deployment do not need to modify their configurations at this time.

3) Monitor the Decommissioning Process

Use the mc admin decommission status command to monitor the decommissioning process.

mc admin decommission status myminio

The command returns output similar to the following:

┌─────┬────────────────────────────────────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID  │ Pools                                                          │ Capacity                         │ Status   │
│ 1st │ https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio │  10 TiB (used) / 10  TiB (total) │ Draining │
│ 2nd │ https://minio-{05...08}.example.com:9000/mnt/disk{1...4}/minio │  95 TiB (used) / 100 TiB (total) │ Pending  │
│ 3rd │ https://minio-{09...12}.example.com:9000/mnt/disk{1...4}/minio │  40 TiB (used) / 500 TiB (total) │ Active   │
│ 4th │ https://minio-{13...16}.example.com:9000/mnt/disk{1...4}/minio │  0  TiB (used) / 500 TiB (total) │ Active   │
└─────┴────────────────────────────────────────────────────────────────┴──────────────────────────────────┴──────────┘

You can retrieve more detailed information by specifying the description of the server pool to the command:

mc admin decommission status myminio https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio

The command returns output similar to the following:

Decommissioning rate at 100MiB/sec [1TiB/10TiB]
Started: 30 minutes ago

mc admin decommission status marks the Status as Complete once decommissioning is completed. You can move on to the next step once MinIO completes decommissioning for all pools.

If Status reads as failed, you can re-run the mc admin decommission start command to resume the process. For persistent failures, use mc admin logs or review the systemd logs (e.g. journalctl -u minio) to identify more specific errors.

4) Remove the Decommissioned Pools from the Deployment Configuration

Once decommissioning completes, you can safely remove the pools from the deployment configuration. Modify the startup command for each remaining MinIO server in the deployment and remove the decommissioned pool.

The .deb or .rpm packages install a systemd service file to /lib/systemd/system/minio.service. For binary installations, this procedure assumes the file was created manually as per the Deploy MinIO: Multi-Node Multi-Drive procedure.

The minio.service file uses an environment file located at /etc/default/minio for sourcing configuration settings, including the startup. Specifically, the MINIO_VOLUMES variable sets the startup command:

cat /etc/default/minio | grep "MINIO_VOLUMES"

The command returns output similar to the following:

MINIO_VOLUMES="https://minio-{1...4}.example.net:9000/mnt/disk{1...4}/minio https://minio-{5...8}.example.net:9000/mnt/disk{1...4}/minio https://minio-{9...12}.example.net:9000/mnt/disk{1...4}/minio"

Edit the environment file and remove the decommissioned pools from the MINIO_VOLUMES value.

5) Update Network Control Plane

Update any load balancers, reverse proxies, or other network control planes to remove the decommissioned server pools from the connection configuration for the MinIO deployment.

Specific instructions for configuring network control plane components is out of scope for this procedure.

6) Restart the MinIO Deployment

Issue the following commands on each node simultaneously in the deployment to restart the MinIO service:

sudo systemctl restart minio.service

Use the following commands to confirm the service is online and functional:

sudo systemctl status minio.service
journalctl -f -u minio.service

MinIO may log an increased number of non-critical warnings while the server processes connect and synchronize. These warnings are typically transient and should resolve as the deployment comes online.

MinIO strongly recommends restarting all MinIO Server processes in a deployment simultaneously. MinIO operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.

Do not perform “rolling” (e.g. one node at a time) restarts.

Once the deployment is online, use mc admin info to confirm the uptime of all remaining servers in the deployment.