`mc batch generate`

Changed in version MinIO: RELEASE.2022-10-08T20-11-00Z or later

Syntax

The mc batch generate command creates a basic YAML-formatted template file for the specified job type.

After MinIO creates the file, open it in your preferred text editor tool to further customize. You can define one job task definition per batch file.

See job types for the supported jobs you can generate.

EXAMPLE

The following command creates a basic YAML file for a replicate job on the mybucket bucket of the myminio alias.

mc batch generate myminio/mybucket replicate

SYNTAX

The command has the following syntax:

mc [GLOBALFLAGS] batch generate \
                       TARGET   \
                       JOBTYPE

Brackets [] indicate optional parameters.
Parameters sharing a line are mutually dependent.
Parameters separated using the pipe | operator are mutually exclusive.

Copy the example to a text editor and modify as-needed before running the command in the terminal/shell.

Parameters

TARGET

Required

The alias used to generate the YAML template file. The specified alias does not restrict the deployment(s) where you can use the generated file.

For example:

mc batch generate myminio replicate

JOBTYPE

Required

The type of job to generate a YAML document for.

Currently, mc batch supports the replicate and keyrotate job types.

Global Flags

This command supports any of the global flags.

Examples

Generate a `yaml` File for a Replicate Job Type

The following command generates a YAML blueprint for a replicate type batch job and names the file replicate with the .yaml extension:

mc batch generate alias replicate > replicate.yaml

Replace alias with the alias to use to generate the yaml file.
Replace replicate with the type of job to generate a yaml file for.

:mc:mc batch supports the replicate and keyrotate job types.

S3 Compatibility

The mc commandline tool is built for compatibility with the AWS S3 API and is tested with MinIO and AWS S3 for expected functionality and behavior.

MinIO provides no guarantees for other S3-compatible services, as their S3 API implementation is unknown and therefore unsupported. While mc commands may work as documented, any such usage is at your own risk.

Job Types

mc batch currently supports the following job task types:

replicate

Replicate objects between two MinIO deployments. Provides similar functionality to bucket replication as a batch job rather than continual scanning function.
keyrotate

New in version MinIO: RELEASE.2023-04-07T05-28-58Z

Rotate the sse-s3 or sse-kms keys for objects at rest on a MinIO deployment.

`replicate`

Use the replicate job type to create a batch job that replicates objects from the local MinIO deployment to another MinIO location.

The YAML must define the source and target deployments. If the _source_ deployment is remote, then the _target_ deployment must be local. Optionally, the YAML can also define flags to filter which objects replicate, send notifications for the job, or define retry attempts for the job.

Changed in version MinIO: RELEASE.2023-04-07T05-28-58Z

You can replicate from a remote MinIO deployment to the local deployment that runs the batch job.

For the source deployment

Required information

type:

Must be minio.

bucket:

The bucket on the deployment.

Optional information

`prefix:`	The prefix on the object(s) that should replicate.
`endpoint:`	Location of the deployment to use for either the source or the target of a replication batch job. For example, `https://minio.example.net`. If the deployment is the mc alias set specified to the command, omit this field to direct MinIO to use that alias for the endpoint and credentials values. Either the source deployment or the remote deployment must be the “local” alias. The non-“local” deployment must specify the `endpoint` and `credentials`.
`path:`	Directs MinIO to use Path or Virtual Style (DNS) lookup of the bucket. - Specify `on` for Path style - Specify `off` for Virtual style - Specify `auto` to let MinIO determine the correct lookup style. Defaults to `auto`.
`credentials:`	The `accesskey:` and `secretKey:` or the `sessionToken:` that grants access to the object(s). Only specify for the deployment that is not the local deployment.

For the target deployment

Required information

type:

Must be minio.

bucket:

The bucket on the deployment.

Optional information

`prefix:`	The prefix on the object(s) to replicate.
`endpoint:`	The location of the target deployment. If the target is the alias specified to the command, you can omit this and the `credentials` fields. If the target is “local”, the source must specify the remote deployment with `endpoint` and `credentials`.
`credentials:`	The `accesskey` and `secretKey` or the `sessionToken` that grants access to the object(s).

For filters

`newerThan:`	A string representing a length of time in `#d#h#s` format. Only objects newer than the specified length of time replicate. For example, `7d`, `24h`, `5d12h30s` are valid strings.
`olderThan:`	A string representing a length of time in `#d#h#s` format. Only objects older than the specified length of time replicate.
`createdAfter:`	A date in `YYYY-MM-DD` format. Only objects created after the date replicate.
`createdBefore:`	A date in `YYYY-MM-DD` format. Only objects created prior to the date replicate.

For notifications

`endpoint:`	The predefined endpoint to send events for notifications.
`token:`	An optional JWT <JSON Web Token> to access the `endpoint`.

For retry attempts

If something interrupts the job, you can define how many attempts to retry the job batch. For each retry, you can also define how long to wait between attempts.

`attempts:`	Number of tries to complete the batch job before giving up.
`delay:`	The least amount of time to wait between each attempt.

Sample YAML

replicate:
  apiVersion: v1
  # source of the objects to be replicated
  # if source is not the local deployment for the command, provide the endpoint and credentials
  source:
    type: TYPE # valid values are "s3" or "minio"
    bucket: BUCKET
    prefix: PREFIX
    # endpoint: ENDPOINT
    # path: "on|off|auto"
    # credentials:
    #   accessKey: ACCESS-KEY
    #   secretKey: SECRET-KEY
    #   sessionToken: SESSION-TOKEN # Available when rotating credentials are used

  # target where the objects must be replicated
  # if target is not the local deployment for the command, provide the endpoint and credentials
  target:
    type: TYPE # valid values are "s3" or "minio"
    bucket: BUCKET
    prefix: PREFIX
    # endpoint: ENDPOINT
    # path: "on|off|auto"
    # credentials:
    #   accessKey: ACCESS-KEY
    #   secretKey: SECRET-KEY
    #   sessionToken: SESSION-TOKEN # Available when rotating credentials are used

  # optional flags based filtering criteria
  # for all source objects
  flags:
    filter:
      newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
      olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
      createdAfter: "date" # match objects created after "date"
      createdBefore: "date" # match objects created before "date"

      # tags:
      #   - key: "name"
      #     value: "pick*" # match objects with tag 'name', with all values starting with 'pick'

      ## NOTE: metadata filter not supported when "source" is non MinIO.
      # metadata:
      #   - key: "content-type"
      #     value: "image/*" # match objects with 'content-type', with all values starting with 'image/'

  notify:
    endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
    token: "Bearer xxxxx" # optional authentication token for the notification endpoint

  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: "500ms" # least amount of delay between each retry

`keyrotate`

New in version MinIO: RELEASE.2023-04-07T05-28-58Z

Use the keyrotate job type to create a batch job that cycles the sse-s3 or sse-kms keys for encrypted objects.

Required information

type:

Either sse-s3 or sse-kms.

key:

Only for use with the sse-kms type. The key to use to unseal the key vault.

context:

Only for use with the sse-kms type. The context within which to perform actions.

Optional information

For flag based filters

`newerThan:`	A string representing a length of time in `#d#h#s` format. Keys rotate only for objects newer than the specified length of time. For example, `7d`, `24h`, `5d12h30s` are valid strings.
`olderThan:`	A string representing a length of time in `#d#h#s` format. Keys rotate only for objects older than the specified length of time.
`createdAfter:`	A date in `YYYY-MM-DD` format. Keys rotate only for objects created after the date.
`createdBefore:`	A date in `YYYY-MM-DD` format. Keys rotate only for objects created prior to the date.
`tags:`	Rotate keys only for objects with tags that match the specified `key:` and `value:`.
`metadtaa:`	Rotate keys only for objects with metadata that match the specified `key:` and `value:`.
`kmskey:`	Rotate keys only for objects with a KMS key-id that match the specified value. This is only applicable for the `sse-kms` type.

For notifications

`endpoint:`	The predefined endpoint to send events for notifications.
`token:`	An optional JWT <JSON Web Token> to access the `endpoint`.

For retry attempts

If something interrupts the job, you can define a maximum number of retry attempts. For each retry, you can also define how long to wait between attempts.

`attempts:`	Number of tries to complete the batch job before giving up.
`delay:`	The amount of time to wait between each attempt.

Sample YAML

keyrotate:
  apiVersion: v1
  bucket: bucket
  prefix: 
  encryption:
    type: sse-kms # valid values are sse-s3 and sse-kms
    
    # The following encryption values only apply for sse-kms type.
    # For sse-s3 key types, MinIO uses the key provided by the MINIO_KMS_KES_KEY_FILE environment variable.
    # The following two values are ignored if type is set to sse-s3.
    key: my-new-keys2 # valid only for sse-kms
    context: <new-kms-key-context> # valid only for sse-kms

  # optional flags based filtering criteria
  flags:
    filter:
      newerThan: "84h" # match objects newer than this value (e.g. 7d10h31s)
      olderThan: "80h" # match objects older than this value (e.g. 7d10h31s)
      createdAfter: "2023-03-02T15:04:05Z07:00" # match objects created after "date"
      createdBefore: "2023-03-02T15:04:05Z07:00" # match objects created before "date"
      tags:
        - key: "name"
          value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
      metadata:
        - key: "content-type"
          value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
      kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms)
  
  # optional entries to add notifications for the job
  notify:
    endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
    token: "Bearer xxxxx" # optional authentication token for the notification endpoint
  
  # optional entries to add retry attempts if the job is interrupted
  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: "500ms" # least amount of delay between each retry

`type:`	Must be `minio`.
`bucket:`	The bucket on the deployment.

`type:`	Either `sse-s3` or `sse-kms`.
`key:`	Only for use with the `sse-kms` type. The key to use to unseal the key vault.
`context:`	Only for use with the `sse-kms` type. The context within which to perform actions.