MinIO supports both synchronous and near-synchronous replication depending on the architectural choices
and rate of change with the data. In each of these scenarios, it is imperative that the replication be as close to strictly consistent as possible (taking into account bandwidth considerations and the rate of change).
At the most basic level any design needs to account for infrastructure, bandwidth, latency,
resilience and scale. Let’s examine them in order:
MinIO recommends the same hardware on both sides of the replication endpoints. While similar hardware will perform, introducing heterogeneous HW profiles introduces complexity and slows issue identification.
Bandwidth is an important factor in keeping both the sites synchronized at all times. The optimal bandwidth requirement between the sites is determined by the rate of incoming data. Specifically, if the bandwidth is not sufficient to handle spikes, then the changes will be queued to remote sites and will eventually synchronize.
After bandwidth, latency is the most important consideration in designing an active-active model. Latency represents the round-trip time (RTT) between the two MinIO clusters. The goal is to drive latency down to the smallest possible figure within the budgetary constraints imposed by bandwidth. MinIO recommends a RTT threshold of no more than 20ms and packet loss of no more than 0.01% for both the Ethernet links and the network.
At present, MinIO only recommends replication across two data centers. It is possible to have replication across multiple data centers, however, the complexity involved and the tradeoffs required make this rather difficult.
MinIO supports very large deployments in each data center, both for source and target,
and the considerations outlined above will dictate scale.
If the target goes down, the source will cache the changes and will start syncing once the replication target comes back up. There may be some delay to reach full sync depending on the length of time, number of changes, bandwidth and latency.
Immutability is supported. Key concepts can be found in this post. In the active-active replication mode, immutability is only guaranteed if the objects are versioned. Versioning cannot be disabled on the source. If versioning is suspended on the target, MinIO will start to fail replication.
In these cases, replication could fail. For example, if you attempt to disable versioning on the source bucket, an error is returned. You must remove the replication configuration before you can disable versioning on the source bucket. Additionally, if you disable versioning on the destination bucket, replication fails.
Object locking must be enabled on both the source and the target. There is a corner case, where, after bucket replication has been set, the target bucket can be deleted and recreated with object lock not enabled, replication can fail. There is a potential for inconsistency if object locking settings are not configured on both ends. MinIO will silently fail in this case.