Batch Replication

Behavior

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetBucketTarget",
                "admin:GetBucketTarget"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteBucketConfiguration"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:PutReplicationConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationRuleConfiguration"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationOnBucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ReplicateTags",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:PutObject",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectLegalHold",
                "s3:DeleteObject",
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicatingDataIntoBucket"
        }
    ]
}

Replicate Batch Job Reference

The YAML must define the source and target deployments. If the source deployment is remote, then the target deployment must be local. Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.

For the source deployment

• Required information

  type:	Must be minio.
  bucket:	The bucket on the deployment.

• Optional information

  prefix:	The prefix on the object(s) that should replicate.
  endpoint:	Location of the deployment to use for either the source or the target of a replication batch job. For example, https://minio.example.net. If the deployment is the mc alias set specified to the command, omit this field to direct MinIO to use that alias for the endpoint and credentials values. Either the source deployment or the remote deployment must be the “local” alias. The non-“local” deployment must specify the endpoint and credentials.
  path:	Directs MinIO to use Path or Virtual Style (DNS) lookup of the bucket. - Specify on for Path style - Specify off for Virtual style - Specify auto to let MinIO determine the correct lookup style. Defaults to auto.
  credentials:	The accesskey: and secretKey: or the sessionToken: that grants access to the object(s). Only specify for the deployment that is not the local deployment.
  snowball	version added: RELEASE.2023-12-09T18-17-51Z Configuration options for controlling the batch-and-compress functionality.
  snowball.disable	Specify true to disable the batch-and-compress functionality during replication. Defaults to false.
  snowball.batch	Specify the maximum integer number of objects to batch for compression. Defaults to 100.
  snowball.inmemory	Specify false to stage archives using local storage or true to stage to memory (RAM). Defaults to true.
  snowball.compress	Specify true to generate compress batched objects over the wire using the S2/Snappy compression algorithm. Defaults to false or no compression.
  snowball.smallerThan	Specify the size of object in Megabits (MiB) under which MinIO should batch objects. Defaults to 5MiB.
  snowball.skipErrs	Specify false to direct MinIO to halt on any object which produces errors on read. Defaults to true.

For the target deployment

• Required information

  type:	Must be minio.
  bucket:	The bucket on the deployment.

• Optional information

  prefix:	The prefix on the object(s) to replicate.
  endpoint:	The location of the target deployment. If the target is the alias specified to the command, you can omit this and the credentials fields. If the target is “local”, the source must specify the remote deployment with endpoint and credentials.
  credentials:	The accesskey and secretKey or the sessionToken that grants access to the object(s).

For filters

For notifications

For retry attempts

If something interrupts the job, you can define how many attempts to retry the job batch. For each retry, you can also define how long to wait between attempts.

Sample YAML Description File for a replicate Job Type

Use mc batch generate to create a basic replicate batch job for further customization.

For the local deployment, do not specify the endpoint or credentials. Either delete or comment out those lines for the source or the target section, depending on which is the local.

replicate:
  apiVersion: v1
  # source of the objects to be replicated
  # if source is not the local deployment for the command, provide the endpoint and credentials
  source:
    type: TYPE # valid values are "s3" or "minio"
    bucket: BUCKET
    prefix: PREFIX
    # endpoint: ENDPOINT
    # path: "on|off|auto"
    # credentials:
    #   accessKey: ACCESS_KEY
    #   secretKey: SECRET_KEY
    #   sessionToken: SESSION_TOKEN # Available when rotating credentials are used
    # snowball:
    #   disable: true|false
    #   batch: 100
    #   inmemory: true|false
    #   compress: true|false
    #   smallerThan: 5MiB
    #   skipErrs: true|false
  # target where the objects must be replicated
  # if target is not the local deployment for the command, provide the endpoint and credentials
  target:
    type: TYPE # valid values are "s3" or "minio"
    bucket: BUCKET
    prefix: PREFIX
    # endpoint: ENDPOINT
    # path: "on|off|auto"
    # credentials:
    #   accessKey: ACCESS_KEY
    #   secretKey: SECRET_KEY
    #   sessionToken: SESSION_TOKEN # Available when rotating credentials are used

  # optional flags based filtering criteria
  # for all source objects
  flags:
    filter:
      newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
      olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
      createdAfter: "date" # match objects created after "date"
      createdBefore: "date" # match objects created before "date"

      # tags:
      #   - key: "name"
      #     value: "pick*" # match objects with tag 'name', with all values starting with 'pick'

      ## NOTE: metadata filter not supported when "source" is non MinIO.
      # metadata:
      #   - key: "content-type"
      #     value: "image/*" # match objects with 'content-type', with all values starting with 'image/'

  notify:
    endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
    token: "Bearer xxxxx" # optional authentication token for the notification endpoint

  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: "500ms" # least amount of delay between each retry