Data Compression
Overview
MinIO Server supports compressing objects to reduce disk usage. Objects are compressed on PUT before writing to disk, and uncompressed on GET before they are sent to the client. This makes the compression process transparent to client applications and services.
Depending on the type of data, compression may also increase overall throughput. Write throughput for a production deployment is generally 500MB per second or greater per available CPU core in the system. Decompression is approximately 1 GB per second or greater for each CPU core.
For best results, review MinIO’s recommended hardware configuration or use MinIO SUBNET to work directly with engineers for analyzing compression performance.
Default File Types
Data compression is a global option, the configured settings apply to all buckets in a deployment. Enabling data compression compresses the following types of data by default:
File Extensions |
Media (MIME) Types |
---|---|
|
|
You can control which objects are compressed by specifying the desired file extensions and media (MIME) types.
Existing objects are not modified
Enabling, disabling, or updating a deployment’s compression settings does not modify existing objects. New objects are compressed according to the settings in effect at the time they are created.
Excluded File Types
Some data cannot be effectively compressed, such as video or already compressed data. MinIO does not compress common incompressible file types, even if they are specified in the compression configuration.
Objects of these types are never compressed:
Object Type |
File Extension |
Media (MIME) Type |
---|---|---|
Audio |
|
|
Video |
*.mp4 *.mkv *.mov |
|
Image |
*.jpg *.png *.gif |
|
7ZIP Compressed |
|
|
BZIP2 Compressed |
|
|
GZIP Compressed |
|
|
RAR Compressed |
|
|
LZMA Compressed |
|
|
ZIP Compressed |
|
application/zip application-x-zip-compressed |
Data Compression and Encryption
MinIO supports encrypting compressed objects but recommends against combining compression and encryption without a prior risk assessment. Before enabling encryption for compressed objects, carefully consider the security needs of your environment.
See Transparent Data Compression on MinIO for more about combining compression and encryption. MinIO SUBNET users can log in and engage with our engineering and security teams to review encryption options.
Tutorials
Enable Data Compression
To enable data compression, use mc admin config set
to set the compression
key enable
option to on
.
The following enables compression for new objects of the default types:
mc admin config set ALIAS compression enable=on
Replace
ALIAS
with thealias
of a configured MinIO deployment.
Existing uncompressed objects are not modified. To configure which extensions and types to compress, see Configure Which Objects to Compress.
To view the current compression settings:
mc admin config get ALIAS compression
Disable Data Compression
To disable data compression, use mc admin config set
to set the compression
key enable
option to off
:
The following disables data compression for new objects:
mc admin config set ALIAS compression enable=off
Replace
ALIAS
with thealias
of a configured MinIO deployment.
Existing compressed objects are not modified.
Configure Which Objects to Compress
Configure the objects to compress by specifying the desired file extensions and media types in extensions
or mime_types
arguments.
The default data compression configuration compresses the following types of data:
File Extensions |
Media (MIME) Types |
---|---|
|
|
Default excluded extensions and types are never compressed
Some objects cannot be efficiently compressed.
MinIO will not attempt to compress these objects, even if they are specified in extensions
or mime_types
arguments.
See Excluded File Types for a list of excluded types.
The sections below describe how to configure compression for the desired file extensions and media types.
Compress All Compressible Objects
To compress all objects except the default excluded types, use mc admin config set
to set the compression
key extensions
and mime_types
options to empty lists:
mc admin config set ALIAS compression extensions= mime_types=
Replace
ALIAS
with thealias
of a configured MinIO deployment.
Compress Objects by File Extension
To compress objects with certain file extensions, use mc admin config set
to set the desired file extensions in an extensions
argument.
The following command compresses files with the extensions .bin
and .txt
:
mc admin config set ALIAS compression extensions=".bin, .txt"
Replace
ALIAS
with thealias
of a configured MinIO deployment.
The new list of file extensions replaces the previous list.
To add or remove an extension, repeat the extensions
command with the complete list of extensions to compress.
The following adds .pdf
to the list of file extensions from the previous example:
mc admin config set ALIAS compression extensions=".bin, .txt, .pdf"
Replace
ALIAS
with thealias
of a configured MinIO deployment.
Compress Objects by Media Type
To compress objects of certain media types, use mc admin config set
to set the compression
key mime_types
option to a list of the desired types.
The following example compresses files of types application/json
and image/bmp
:
mc admin config set ALIAS compression mime_types="application/json, image/bmp"
Replace
ALIAS
with thealias
of a configured MinIO deployment.
The new list of media types replaces the previous list.
To add or remove a type, repeat the mime_types
command with the complete list of types to compress.
You can use *
to specify all subtypes of a single media type.
The following command adds all text
subtypes to the list from the previous example:
mc admin config set ALIAS compression mime_types="application/json, image/bmp, text/*"
Replace
ALIAS
with thealias
of a configured MinIO deployment.