Enable Multi-Site Server-Side Bucket Replication
The procedure on this page configures automatic server-side bucket replication between multiple MinIO deployments. Multi-Site Active-Active replication builds on the Enable Two-Way Server-Side Bucket Replication procedure with additional considerations required to ensure predictable replication behavior across all sites.
To configure replication between arbitrary S3-compatible services, use
mc mirror
.To configure one-way “active-active” replication between two MinIO deployments, see Enable Two-Way Server-Side Bucket Replication.
To configure one-way “active-passive” replication between MinIO deployments, see Enable One-Way Server-Side Bucket Replication.
Multi-Site Active-Active replication configurations can span multiple racks, datacenters, or geographic locations. Complexity of configuring and maintaining multi-site configurations generally increase with the number of sites and size of each site. Enterprises looking to implement multi-site replication should consider leveraging MinIO SUBNET support to access the expertise, planning, and engineering resources required for addressing that use case.
See also
Use the
mc replicate update
command to modify an existing replication rule.Use the
mc replicate update
command with the--state "disable"
flag to disable an existing replication rule.Use the
mc replicate rm
command to remove an existing replication rule.
Requirements
You must meet all of the basic requirements for bucket replication described in Bucket Replication Requirements.
In addition, to create multi-site bucket replication set up, you must meet the following additional requirements:
Access to All Clusters
You must have network access and log in credentials with correct permissions to all deployments to set up multi-site active-active bucket replication.
You can access the deployments by logging in to the MinIO Console for each deployment or by installing mc
and using the command line.
If using the command line, use the mc alias set
command to create an alias for each MinIO deployment.
Alias creation requires specifying an access key for a user on the deployment.
This user must have permission to create and manage users and policies on the deployment.
Specifically, ensure the user has at minimum:
Considerations
Click to expand any of the following:
Use Consistent Replication Settings
MinIO supports customizing the replication configuration to enable or disable the following replication behaviors:
Replication of delete operations
Replication of delete markers
Replication of existing objects
Replication of metadata-only changes
When configuring replication rules for a bucket, ensure that all MinIO deployments participating in multi-site replication use the same replication behaviors to ensure consistent and predictable synchronization of objects.
Replication of Existing Objects
MinIO supports automatically replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the mc replicate add --replicate
or mc replicate update --replicate
and including the existing-objects
replication feature flag.
This procedure includes the required flags for enabling replication of existing objects.
Replication of Delete Operations
MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate versioning Delete Markers and the deletion of specific versioned objects:
For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
MinIO requires explicitly enabling replication of delete operations using the mc replicate add --replicate
or mc replicate update --replicate
.
This procedure includes the required flags for enabling replication of delete operations and delete markers.
MinIO does not replicate delete operations resulting from the application of lifecycle management expiration rules. Configure matching expiration rules for the bucket on all replication sites to ensure consistent application of object expiration.
Procedure
This procedure requires repeating steps for each MinIO deployment participating in the multi-site replication configuration. Depending on the number of deployments, this procedure may require significant time and care in implementation. MinIO recommends reading through the procedure before attempting to implement the documented steps.
Configure Multi-Site Bucket Replication Using the MinIO Console
1) Create the Replication Rules
Log in to the MinIO Console for the deployment
Select the Manage button for the bucket to replicate
Select the Replication section
Select Add Replication Rule +
Complete the requested information:
Field
Description
Priority
Enter a number value to indicate the order in which to process replication rules for the bucket. 1 indicates the highest importance.
Target URL
The URL of the deployment to replicate data to.
Use TLS
Leave the toggle in the ON position if the destination deployment uses TLS. Otherwise, move the toggle to the OFF position.
Access Key
The user name to use on the destination deployment. The user must have write access to the bucket to replicate to.
Secret Key
The password for the provided Access Key.
Target Bucket
The bucket at the destination to write the data to. The target bucket may have the same name as the origin bucket, depending on the destination bucket location.
Region
The AWS resource region location of the destination deployment.
Replication mode
Leave the default selection of Asynchronous to allow MinIO to replicate data after the write operation completes on the origin ment. Select Synchronous to attempt to complete the replication of the object during its write operation.
While synchronous replication may result in more reliable synchronization between the origin and destination buckets, it may also increase the time of each write operation.
Bandwidth
Specify the maximum amount of bandwidth the replication process can use while replicating data. Enter a number and select a data unit.
Health Check Duration
The maximum length of time in seconds MinIO should spend verifying the health of the replicated data on the destination bucket.
Storage Class
The class of storage to use on the destination deployment for the replicated data. Valid values are either
STANDARD
orREDUCED_REDUNDANCY
.Object Filters
Limit which objects to replicate from the bucket by Prefix or tags. If you enter multiple tags, the objects must match all tag values.
Metadata Sync
Leave selected to also replicate the object’s metadata file. Otherwise, move the toggle to the Off position.
Delete Markers
Leave selected to also replicate MinIO’s indication that an object has been deleted and should also be marked deleted at the action bucket. Otherwise, move the toggle to the Off position to prevent marking the object as deleted in the destination bucket.
Deletes
Leave selected to allow replication of the deletion of versions of an object. Otherwise, move the toggle to the Off position to not replicate deletion of object versions.
Select Save to finish adding the replication rule
Repeat the above steps to create a rule from this deployment to each of the other target deployments.
Then, repeat the above steps on each of the other deployments in the multi-site setup so that each deployment has a separate replication rule for all of the other deployments.
2) Validate the Replication Configuration
Go to the Buckets section of the MinIO Console
Select the Browse button for the bucket you added replication to
Select the Upload button to add a new object to the bucket
Select Upload File
Use the interface to add a new object to the bucket
Go to the other deployment’s console and select the destination bucket defined in the replication
Repeat this test on each deployment by copying a new unique file and checking that the file replicates to each of the other deployments.
Configure Multi-Site Bucket Replication Using the Command Line mc
This procedure uses the placeholder ALIAS
to reference the alias each MinIO deployment being configured for replication.
Replace these values with the appropriate alias for each MinIO deployment.
This procedure assumes each alias corresponds to a user with the necessary replication permissions.
Changed in version RELEASE.2022-12-24T15-21-38Z: mc replicate add
automatically creates the necessary replication targets, removing the need for using the deprecated mc admin remote bucket add
command.
This procedure only documents the procedure as of that release.
1) Create New Bucket Replication Rules
Use the mc replicate add
command to add a new replication rule to each MinIO deployment.
mc replicate add ALIAS/BUCKET \
--remote-bucket 'https://USER:PASSWORD@HOSTNAME:PORT/BUCKET' \
--replicate "delete,delete-marker,existing-objects"
Replace
ALIAS
with the alias of the origin MinIO deployment. The name must match the bucket specified when creating the remote target in the previous step.Replace
BUCKET
with the name of the bucket to replicate from on the origin deployment.Replace the
--remote-bucket
to specify the remote MinIO deployment and bucket to which theALIAS/BUCKET
replicates.The
USER:PASSWORD
must correspond to a user on the remote deployment with the necessary replication permissions.The
HOSTNAME:PORT
must resolve to a reachable MinIO instance on the remote deployment. TheBUCKET
must exist and otherwise meet all other replication requirements.The
--replicate "delete,delete-marker,existing-objects"
flag enables the following replication features:See
mc replicate add --replicate
for more complete documentation. Omit any field to disable replication of that component.
Specify any other supported optional arguments for mc replicate add
.
Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration.
For example, a multi-site replication configuration consisting of MinIO deployments minio1
, minio2
, and minio3
would require repeating this step on each deployment for each remote.
Specifically, in this scenario, perform this step twice on each deployment:
On the
minio1
deployment, once for a rule forminio2
and again for a separate rule forminio3
.On the
minio2
deployment, once for a rule forminio1
and again for a separate rule forminio3
.On the
minio3
deployment, once for a rule forminio1
and again for a separate rule forminio2
.
2) Validate the Replication Configuration
Use mc cp
to copy a new object to the replicated bucket on one of the deployments.
mc cp ~/foo.txt ALIAS/BUCKET
Use mc ls
to verify the object exists on the destination bucket:
mc ls ALIAS/BUCKET
Repeat this test on each deployment by copying a new unique file and checking that the file replicates to each of the other deployments.
You can also use mc stat
to check the file to check the current replication stage of the object.