Batch Expiration

New in version MinIO: RELEASE.2023-12-02T10-51-33Z

The MinIO Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the MinIO Client.

The expire batch job applies Automatic Object Expiration behavior to a single bucket. The job determines expiration eligibility based on the provided configuration, independent of any configured expiration rules.

Behavior

Immediate Expiration of Objects

Batch expiration occurs immediately as part of the batch job, as compared to the passive scanner-based application of expiration rules. Specifically, batch expiration does not yield to application I/O and may impact performance of regular read/write operations on the deployment.

Expiration Eligibility Determined at Batch-Run

The batch expiration works per-bucket and runs once to completion. The job determines expiration eligibility at the time the job runs, and does not rescan or recheck for new objects periodically.

To capture any new objects eligible for expiration, re-run the batch job.

Expiry Rules Check Latest Object Only

The batch expiration job only checks the latest or “current” version of each object against each batch expiration rule.

Expire Batch Job Reference

Field	Description
`expire`	Required Top-level field for the expiration job type.
`apiVersion`	Required Set to `v1`.
`bucket`	Required Specify the name of the bucket in which the job runs.
`prefix`	Optional Specify the bucket prefix in which the job runs.
`rules`	Required An array of one or more expiration rules to apply to objects in the specified `bucket` and `prefix` (if any).
`rules.[n].type`	Required Supports one of the following two values: `object` - Applies only to objects which do not have a `DeleteMarker` as the current version. `deleted` - Applies only to objects which do Have a `DeleteMarker` as the current version. See Object Deletion for more complete documentation on `DeleteMarker` or delete operations in versioned buckets.
`rules.[n].name`	Optional Specify a match string to use for filtering objects. Supports glob-style wildcards (`*`, `?`).
`rules.[n].olderThan`	Optional Specify the age of objects for filtering objects. The rule applies to only those objects older than the specified unit of time. For example, `72h` or `3d` selects objects older than three days.
`rules.[n].createdBefore`	Optional Specify an ISO-8601 timestamp for filtering objects. The rule applies to only those objects created before the specified timestamp.
`rules.[n].tags`	Optional Specify an array of key-value pairs describing object tags to use for filtering objects. The `value` entry supports glob-style wildcards (`*`, `?`). For example, the following filters the rule to only objects with matching tags: tags: - key: archive value: True This key is incompatible with `rules.[n].type: deleted`.
`rules.[n].metadata`	Optional Specify an array of key-value pairs describing object metadata to use for filtering objects. The `value` key supports glob-style wildcards (``, `?`). For example, the following filters the rule to only objects with matching metadata: metadata: - key: content-type value: image/ This key is incompatible with `rules.[n].type: deleted`.
`rules.[n].size`	Optional Specify the range of object sizes for filtering objects. `lessThan` - matches objects with size less than the specified amount (e.g. `MiB`, `GiB`). `greaterThan` - matches objects with size greater than the specified amount (e.g. `MiB`, `GiB`).
`rules.[n].purge.retainVersions`	Optional Specify the number of object versions to retain when applying expiration. Defaults to `0` for deleting all object versions (fastest).
`notify.endpoint`	Optional The predefined endpoint to send events for notifications.
`notify.token`	Optional An optional JSON Web Token (JWT) to access the `notify.endpoint`.
`retry.attempts`	Optional The number of tries to complete the batch job before giving up.
`retry.delay`	Optional The amount of time to wait between each attempt (`ms`).

Sample YAML Description for an `expire` Job Type

Use mc batch generate to create a basic expire batch job for further customization.

expire:
  apiVersion: v1
  bucket: mybucket # Bucket where this job will expire matching objects from
  prefix: myprefix # (Optional) Prefix under which this job will expire objects matching the rules below.
  rules:
    - type: object  # objects with zero ore more older versions
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 70h # match objects older than this value
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      tags:
        - key: name
          value: pick* # match objects with tag 'name', all values starting with 'pick'
      metadata:
        - key: content-type
          value: image/* # match objects with 'content-type', all values starting with 'image/'
      size:
        lessThan: 10MiB # match objects with size less than this value (e.g. 10MiB)
        greaterThan: 1MiB # match objects with size greater than this value (e.g. 1MiB)
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object.

    - type: deleted # objects with delete marker as their latest version
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 10h # match objects older than this value (e.g. 7d10h31s)
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object including delete markers.

  notify:
    endpoint: https://notify.endpoint # notification endpoint to receive job completion status
    token: Bearer xxxxx # optional authentication token for the notification endpoint

  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: 500ms # least amount of delay between each retry