Replication

Replication, and specifically active-active, multisite replication is a key requirement for mission-critical production environments. MinIO is the only vendor that offers it today and does so with bucket level granularity.

Active-Active Replication

MinIO supports both synchronous and near-synchronous replication depending on the architectural choice and rate of change with the data. In each of these scenarios, it is imperative that the replication be as close to strictly consistent as possible (taking into account bandwidth considerations and the rate of change).

MinIO's approach to data replication creates active-active storage systems that can be used on:

Encrypted or unencrypted objects and their associated metadata (written atomically with the object).
Object versions.
Object tags, if there are any.
S3 Object Lock retention information, if there is any.

MinIO’s data replication is designed for resilience at scale.
Key features include:

The ability for source and destination buckets to have the same name. This is a requirement for applications that must transparently failover to the remote site without any disruption.
Native support for automatic object locking/retention replication across the source and destination.
Near-synchronous replication to update objects immediately after any mutation on the bucket. MinIO follows strict consistency within the data center and eventual-consistency across data centers to protect the data.
Notification functionality to push replication failure events. Applications can subscribe to these events and alert the operations team.

Multi-Site, Active-Active Replication for Object Storage

What to Consider When
Implementing MinIO’s
Active-Active Replication

At the most basic level any design needs to account for infrastructure, bandwidth, latency, resilience and scale. Let’s examine them in order:

Infrastructure

MinIO recommends the same hardware on both sides of the replication endpoints. While similar hardware will perform, introducing heterogeneous HW profiles introduces complexity and slows issue identification.
Bandwidth

Bandwidth is an important factor in keeping both the sites synchronized at all times. The optimal bandwidth requirement between the sites is determined by the rate of incoming data. Specifically, if the bandwidth is not sufficient to handle spikes, then the changes will be queued to remote sites and will eventually synchronize.
Latency

After bandwidth, latency is the most important consideration in designing an active-active model. Latency represents the round-trip time (RTT) between the two MinIO clusters. The goal is to drive latency down to the smallest possible figure within the budgetary constraints imposed by bandwidth. MinIO recommends a RTT threshold of no more than 20ms and packet loss of no more than 0.01% for both the Ethernet links and the network.
Architecture

At present, MinIO only recommends replication across two data centers. It is possible to have replication across multiple data centers, however, the complexity involved and the tradeoffs required make this rather difficult.

MinIO supports very large deployments in each data center, both for source and target, and the considerations outlined above will dictate scale.

Multi-Site, Active-Active Replication for Object Storage

FAQ

What happens when the replication target goes down?

If the target goes down, the source will cache the changes and will start syncing once the replication target comes back up. There may be some delay to reach full sync depending on the length of time, number of changes, bandwidth and latency.

What are the parameters on immutability?

Immutability is supported. Key concepts can be found in this post. In the active-active replication mode, immutability is only guaranteed if the objects are versioned. Versioning cannot be disabled on the source. If versioning is suspended on the target, MinIO will start to fail replication.

What are the other implications if versioning is suspended or there is a mismatch?

In these cases, replication could fail. For example, if you attempt to disable versioning on the source bucket, an error is returned. You must remove the replication configuration before you can disable versioning on the source bucket. Additionally, if you disable versioning on the destination bucket, replication fails.

How is object locking handled if it is not enabled on both sides?

Object locking must be enabled on both the source and the target. There is a corner case, where, after bucket replication has been set, the target bucket can be deleted and recreated with object lock not enabled, replication can fail. There is a potential for inconsistency if object locking settings are not configured on both ends. MinIO will silently fail in this case.

Multi-Site, Active-Active Replication for Object Storage