mc batch generate
Changed in version MinIO: RELEASE.2022-10-08T20-11-00Z or later
Syntax
The mc batch generate
command creates a basic YAML-formatted template file for the specified job type.
After MinIO creates the file, open it in your preferred text editor tool to further customize. You can define one job task definition per batch file.
See job types for the supported jobs you can generate.
The following command creates a basic YAML file for a replicate job on the mybucket
bucket of the myminio
alias.
mc batch generate myminio/mybucket replicate
The command has the following syntax:
mc [GLOBALFLAGS] batch generate \
TARGET \
JOBTYPE
Brackets
[]
indicate optional parameters.Parameters sharing a line are mutually dependent.
Parameters separated using the pipe
|
operator are mutually exclusive.
Copy the example to a text editor and modify as-needed before running the command in the terminal/shell.
Parameters
- TARGET
- Required
The alias used to generate the YAML template file. The specified
alias
does not restrict the deployment(s) where you can use the generated file.For example:
mc batch generate myminio replicate
Global Flags
This command supports any of the global flags.
Examples
Generate a yaml
File for a Replicate Job Type
The following command generates a YAML blueprint for a replicate type batch job and names the file replicate
with the .yaml
extension:
mc batch generate alias replicate > replicate.yaml
Replace
alias
with thealias
to use to generate the yaml file.Replace
replicate
with the type of job to generate a yaml file for.:mc:
mc batch
supports thereplicate
andkeyrotate
job types.
S3 Compatibility
The mc commandline tool is built for compatibility with the AWS S3 API and is tested with MinIO and AWS S3 for expected functionality and behavior.
MinIO provides no guarantees for other S3-compatible services, as their S3 API implementation is unknown and therefore unsupported. While mc commands may work as documented, any such usage is at your own risk.
Job Types
mc batch
currently supports the following job task types:
-
Replicate objects between two MinIO deployments. Provides similar functionality to bucket replication as a batch job rather than continual scanning function.
-
New in version MinIO: RELEASE.2023-04-07T05-28-58Z
Rotate the sse-s3 or sse-kms keys for objects at rest on a MinIO deployment.
-
New in version MinIO: RELEASE.2023-12-02T10-51-33Z
Expire objects based using similar semantics as Automatic Object Expiration.
replicate
You can use the following example configuration as the starting point for building your own custom replication batch job:
replicate:
apiVersion: v1
# source of the objects to be replicated
# if source is not the local deployment for the command, provide the endpoint and credentials
source:
type: TYPE # valid values are "s3" or "minio"
bucket: BUCKET
prefix: PREFIX
# endpoint: ENDPOINT
# path: "on|off|auto"
# credentials:
# accessKey: ACCESS_KEY
# secretKey: SECRET_KEY
# sessionToken: SESSION_TOKEN # Available when rotating credentials are used
# snowball:
# disable: true|false
# batch: 100
# inmemory: true|false
# compress: true|false
# smallerThan: 5MiB
# skipErrs: true|false
# target where the objects must be replicated
# if target is not the local deployment for the command, provide the endpoint and credentials
target:
type: TYPE # valid values are "s3" or "minio"
bucket: BUCKET
prefix: PREFIX
# endpoint: ENDPOINT
# path: "on|off|auto"
# credentials:
# accessKey: ACCESS_KEY
# secretKey: SECRET_KEY
# sessionToken: SESSION_TOKEN # Available when rotating credentials are used
# optional flags based filtering criteria
# for all source objects
flags:
filter:
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "date" # match objects created after "date"
createdBefore: "date" # match objects created before "date"
# tags:
# - key: "name"
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
## NOTE: metadata filter not supported when "source" is non MinIO.
# metadata:
# - key: "content-type"
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry
See Replicate Batch Job Reference for more complete documentation on each key.
keyrotate
You can use the following example configuration as the starting point for building your own custom key rotation batch job:
keyrotate:
apiVersion: v1
bucket: bucket
prefix:
encryption:
type: sse-kms # valid values are sse-s3 and sse-kms
# The following encryption values only apply for sse-kms type.
# For sse-s3 key types, MinIO uses the key provided by the MINIO_KMS_KES_KEY_FILE environment variable.
# The following two values are ignored if type is set to sse-s3.
key: my-new-keys2 # valid only for sse-kms
context: <new-kms-key-context> # valid only for sse-kms
# optional flags based filtering criteria
flags:
filter:
newerThan: "84h" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "80h" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "2023-03-02T15:04:05Z07:00" # match objects created after "date"
createdBefore: "2023-03-02T15:04:05Z07:00" # match objects created before "date"
tags:
- key: "name"
value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
metadata:
- key: "content-type"
value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms)
# optional entries to add notifications for the job
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
# optional entries to add retry attempts if the job is interrupted
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry
See Key Rotate Batch Job Reference for more complete documentation on each key.
expire
You can use the following example configuration as a starting point for building your own custom expiration batch job:
keyrotate:
apiVersion: v1
bucket: bucket
prefix:
encryption:
type: sse-kms # valid values are sse-s3 and sse-kms
# The following encryption values only apply for sse-kms type.
# For sse-s3 key types, MinIO uses the key provided by the MINIO_KMS_KES_KEY_FILE environment variable.
# The following two values are ignored if type is set to sse-s3.
key: my-new-keys2 # valid only for sse-kms
context: <new-kms-key-context> # valid only for sse-kms
# optional flags based filtering criteria
flags:
filter:
newerThan: "84h" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "80h" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "2023-03-02T15:04:05Z07:00" # match objects created after "date"
createdBefore: "2023-03-02T15:04:05Z07:00" # match objects created before "date"
tags:
- key: "name"
value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
metadata:
- key: "content-type"
value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms)
# optional entries to add notifications for the job
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
# optional entries to add retry attempts if the job is interrupted
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry
See Expire Batch Job Reference for more complete documentation on each key.