Server-Side Object Encryption with AWS Secrets Manager Root KMS

MinIO Server-Side Encryption (SSE) protects objects as part of write operations, allowing clients to take advantage of server processing power to secure objects at the storage layer (encryption-at-rest). SSE also provides key functionality to regulatory and compliance requirements around secure locking and erasure.

MinIO SSE uses Key Encryption Service (KES) and an external root Key Management Service (KMS) for performing secured cryptographic operations at scale. The root KMS provides stateful and secured storage of External Keys (EK) while KES is stateless and derives additional cryptographic keys from the root-managed EK.

This procedure assumes a single host machine running the MinIO and KES containers, with AWS Secrets Manager as the external root KMS.. As part of this procedure, you will:

Deploy a KES container configured to use AWS Secrets Manager as the root KMS.
Create a new EK on Vault for use with SSE.
Deploy a MinIO Server container in Single-Node Single-Drive mode configured to use the KES container for supporting SSE.
Configure automatic bucket-default SSE-KMS.

For production orchestrated environments, use the MinIO Kubernetes Operator to deploy a tenant with SSE enabled and configured for use with AWS Key Management Service.

For production baremetal environments, see the MinIO on Linux documentation for tutorials on configuring MinIO with KES and AWS Key Management Service.

Important

Enabling SSE on a MinIO deployment automatically encrypts the backend data for that deployment using the default encryption key.

MinIO requires access to KES and the root KMS to decrypt the backend and start normally. You cannot disable KES later or “undo” the SSE configuration at a later point.

Prerequisites

Ensure Access to the AWS Secrets Manager and Key Management Service

This procedure assumes access to and familiarity with AWS Secrets Manager and AWS Key Management Service.

MinIO specifically requires the following AWS settings or configurations:

A new AWS Programmatic Access user with corresponding access key and secret key.

A policy that grants the created user access to AWS Secrets Manager and AWS Key Management Service. The following policy grants the minimum necessary permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "minioSecretsManagerAccess",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:DeleteSecret",
        "secretsmanager:GetSecretValue",
        "secretsmanager:ListSecrets"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "minioKmsAccess",
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:Encrypt"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

AWS provides the SecretsManagerReadWrite and AWSKeyManagementServicePowerUser canned roles that meet and exceed the minimum required permissions.

Install Podman or a Similar Container Management Interface

This procedure assumes you have a working Podman installation configured to run in “Rootfull” mode.

“Rootless” modes may not provide sufficient permissions to run KES with the necessary security settings. See the relevant “rootless” documentation for more information.

(Podman) Deploy MinIO and KES with Server-Side Encryption using AWS Secrets Manager

Prior to starting these steps, create the following folders:

mkdir -P ~/minio-kes-aws/certs
mkdir -P ~/minio-kes-aws/config
mkdir -P ~/minio-kes-aws/minio

For Windows hosts, substitute the paths with Windows-style paths, e.g. C:\minio-kes-vault\.

1) Generate TLS Certificates for KES and MinIO

The following commands create two TLS certificates that expire within 30 days of creation:

A TLS certificate for KES to secure communications between it and the AWS Secrets Manager service.
A TLS certificate for MinIO to perform mTLS authentication to KES.

Use Caution in Production Environments

DO NOT use the TLS certificates generated as part of this procedure for any long-term development or production environments.

Defer to organization/industry best practices around TLS certificate generation and management. A complete guide to creating valid certificates (e.g. well-formed, current, and trusted) is beyond the scope of this procedure.

# These commands output keys to ~/minio-kes-aws/certs and ~/minio-kes-aws/certs on the host operating system

podman run --rm  \
  -v ~/minio-kes-aws/certs:/certs  \
  quay.io/minio/kes:2024-01-11T13-09-29Z identity new  kes_server \
    --key  /certs/kes-server.key  \
    --cert /certs/kes-server.cert  \
    kes-server

podman run --rm  \
  -v ~/minio-kes-aws/certs:/certs  \
  quay.io/minio/kes:2024-01-11T13-09-29Z identity new minio_server \
    --key  /certs/minio-kes.key  \
    --cert /certs/minio-kes.cert  \
    minio-server

Depending on your Vault configuration, you may need to pass the kes-server.cert as a trusted Certificate Authority. See the Hashicorp Vault Configuration Docs for more information. Defer to the client documentation for instructions on trusting a third-party CA.

2) Create the KES and MinIO Configurations

Create the KES Configuration File

Create the configuration file using your preferred text editor. The following example uses nano:

nano ~/minio-kes-aws/config/kes-config.yaml

KES uses a YAML-formatted configuration file. The following example YAML specifies the minimum required fields for enabling SSE using AWS Secrets Manager:

address: 0.0.0.0:7373

# Disable the root identity, as we do not need that level of access for
# supporting SSE operations.
root: disabled

# Specify the TLS keys generated in the previous step here
# For production environments, use keys signed by a known and trusted
# Certificate Authority (CA).
tls:
  key:  /certs/kes-server.key
  cert: /certs/kes-server.cert

# Create a policy named 'minio' that grants access to the
# /create, /generate, and /decrypt KES APIs for any key name
# KES uses mTLS to grant access to this policy, where only the client
# whose TLS certificate hash matches one of the "identities" can
# use this policy. Specify the hash of the MinIO server TLS certificate
# hash here.
policy:
  minio:
    allow:
    - /v1/key/create/*   # You can replace these wildcard '*' with a string prefix to restrict key names
    - /v1/key/generate/* # e.g. '/minio-'
    - /v1/key/decrypt/*
    - /v1/key/bulk/decrypt
    - /v1/key/list/*
    - /v1/status
    - /v1/metrics
    - /v1/log/audit
    - /v1/log/error
    identities:
    - ${MINIO_IDENTITY_HASH} # Replace with the output of 'kes identity of minio-kes.cert'

                             # In production environments, each client connecting to KES must
                             # Have their TLS hash listed under at least one `policy`.

# Specify the connection information for the KMS and Secrets Manager endpoint.
# The endpoint should be resolvable from the host.
# This example assumes that the associated AWS account has the necessary
# access key and secret key
keystore:
  aws:
    secretsmanager:
      endpoint: secretsmanager.REGION.amazonaws.com # use the Secrets Manager endpoint for your region
      region: REGION # e.g. us-east-1
      kmskey: "" # Optional. The root AWS KMS key to use for cryptographic operations. Formerly described as the "Customer Master Key".
      credentials:
        accesskey: "AWSACCESSKEY" # AWS Access Key
        secretkey: "AWSSECRETKEY" # AWS Secret Key

Set MINIO_IDENTITY_HASH to the identity hash of the MinIO mTLS certificate.

The following command computes the necessary hash:

podman run --rm                                             \
   -v ~/minio-kes-aws/certs/certs:/certs                                \
   kes:2024-01-11T13-09-29Z tool identity of /certs/minio-kes.cert

Set MINIO_IDENTITY_HASH to the identity hash of the MinIO mTLS certificate.

The following command computes the necessary hash:

podman run --rm                                             \
   -v ~/minio-kes-aws/certs/certs:/certs                                \
   kes:2024-01-11T13-09-29Z tool identity of /certs/minio-kes.cert

Replace the REGION with the appropriate region for AWS Secrets Manager. The value must match for both endpoint and region.
Set AWSACCESSKEY and AWSSECRETKEY to the appropriate AWS Credentials.

Create the MinIO Environment File

Create the environment file using your preferred text editor. The following example uses nano:
```
nano ~/minio-kes-aws/config/minio
```
This command assumes the minio-kes.cert, minio-kes.key, and kes-server.cert certificates are accessible at the specified location:
```
MINIO_ROOT_USER=myminioadmin
MINIO_ROOT_PASSWORD=minio-secret-key-change-me
MINIO_VOLUMES="/mnt/data"

# KES Configurations

MINIO_KMS_KES_ENDPOINT=https://127.0.0.1:7373
MINIO_KMS_KES_CERT_FILE=/certs/minio-kes.cert
MINIO_KMS_KES_KEY_FILE=/certs/minio-kes.key
MINIO_KMS_KES_CAPATH=/certs/server.cert
MINIO_KMS_KES_KEY_NAME=minio-backend-default-key
MINIO_KMS_KES_ENCLAVE=<name>
```
MinIO uses the MINIO_KMS_KES_KEY_NAME key for the following cryptographic operations:
- Encrypting the MinIO backend (IAM, configuration, etc.)
- Encrypting objects using SSE-KMS if the request does not include a specific EK.
- Encrypting objects using SSE-S3.
MinIO uses the MINIO_KMS_KES_ENCLAVE key to define the name of the KES enclave to use.
- Replace <name> with the name of the enclave to use.
- If not defined, MinIO does not send any enclave information. This may result in using the default enclave for stateful KES servers.
  
  A KES enclave isolates its associated keys from other enclaves on a stateful KES server.
The minio-kes certificates enable for mTLS between the MinIO deployment and the KES server only. They do not otherwise enable TLS for other client connections to MinIO.

KES automatically creates this key if it does not already exist on the root KMS.

3) Create Pod and Containers

The commands in this section create the following resources:

A Podman Pod to facilitate container communications
A Container for the KES Server configured to use AWS Secrets Manager as the Root KMS.
A Container for a MinIO Server running in Single-Node Single-Drive Mode.

sudo podman pod create  \
  -p 9000:9000 -p 9001:9001 -p 7373:7373  \
  -v ~/minio-kes-aws/certs:/certs  \
  -v ~/minio-kes-aws/minio:/mnt/minio  \
  -v ~/minio-kes-aws/config:/etc/default/  \
  -n minio-kes-aws

sudo podman run -dt  \
  --cap-add IPC_LOCK  \
  --name kes-server  \
  --pod "minio-kes-aws"  \
  -e KES_SERVER=https://127.0.0.1:7373  \
  -e KES_CLIENT_KEY=/certs/kes-server.key  \
  -e KES_CLIENT_CERT=/certs/kes-server.cert  \
  quay.io/minio/kes:2024-01-11T13-09-29Z server  \
    --auth  \
    --config=/etc/default/kes-config.yaml  \

sudo podman run -dt  \
  --name minio-server  \
  --pod "minio-kes-aws"  \
  -e "MINIO_CONFIG_ENV_FILE=/etc/default/minio"  \
  quay.io/minio/minio:RELEASE.2024-02-17T01-15-57Z server  \
    --console-address ":9001"

You can verify the status of the containers using the following commands:

# Should show three pods - one for the Pod, one for KES, and one for MinIO
sudo podman container ls

If all pods are operational, you can connect to the MinIO deployment by opening your browser to http://127.0.0.1:9000 and logging in with the root credentials specified in the MinIO environment file.

4) Generate a New Encryption Key

Unseal Vault Before Creating Key

You must unseal the backing Vault instance before creating new encryption keys. See the Vault documentation on Seal/Unseal for more information.

MinIO requires that the EK exist on the root KMS before performing SSE operations using that key. Use kes key create or mc admin kms key create to create a new EK for use with SSE.

The following command uses the kes key create command to add a new External Key (EK) stored on the root KMS server for use with encrypting the MinIO backend.

sudo podman run --rm  \
  -v ~/minio-kes-aws/certs:/certs  \
  -e KES_SERVER=https://127.0.0.1:7373  \
  -e KES_CLIENT_KEY=/certs/minio-kes.key  \
  -e KES_CLIENT_CERT=/certs/minio-kes.cert  \
  kes:2024-01-11T13-09-29Z key create -k my-new-encryption-key

You can specify any key name as appropriate for your use case, such as a bucket-specific key minio-mydata-key.

5) Enable SSE-KMS for a Bucket

You can use either the MinIO Console or the MinIO mc CLI to enable bucket-default SSE-KMS with the generated key:

MinIO Console

Open the MinIO Console by navigating to http://127.0.0.1:9001 in your preferred browser and logging in with the root credentials specified to the MinIO container.

Once logged in, create a new Bucket and name it to your preference. Select the Gear icon to open the management view.

Select the pencil icon next to the Encryption field to open the modal for configuring a bucket default SSE scheme.

Select SSE-KMS, then enter the name of the key created in the previous step.

Once you save your changes, try to upload a file to the bucket. When viewing that file in the object browser, note that in the sidebar the metadata includes the SSE encryption scheme and information on the key used to encrypt that object. This indicates the successful encrypted state of the object.

MinIO CLI

The following commands:

Create a new alias for the MinIO deployment
Create a new bucket for storing encrypted data
Enable SSE-KMS encryption on that bucket

mc alias set local http://127.0.0.1:9000 ROOTUSER ROOTPASSWORD

mc mb local/encryptedbucket
mc encrypt set SSE-KMS encrypted-bucket-key ALIAS/encryptedbucket

Write a file to the bucket using mc cp or any S3-compatible SDK with a PutObject function. You can then run mc stat on the file to confirm the associated encryption metadata.

Configuration Reference for AWS Root KMS

The following section describes each of the Key Encryption Service (KES) configuration settings for using AWS Secrets Manager and AWS Key Management System as the root KMS for SSE:

Important

Starting with https://github.com/minio/minio/releases/tag/RELEASE.2023-02-17T17-52-43Z, MinIO requires expanded KES permissions for functionality. The example configuration in this section contains all required permissions.

YAML Overview

Fields with ${<STRING>} use the environment variable matching the <STRING> value. You can use this functionality to set credentials without writing them to the configuration file.

The YAML assumes a minimal set of permissions for the MinIO deployment accessing KES. As an alternative, you can omit the policy.minio-server section and instead set the ${MINIO_IDENTITY} hash as the ${ROOT_IDENTITY}.

address: 0.0.0.0:7373
root: ${ROOT_IDENTITY}

tls:
  key: kes-server.key
  cert: kes-server.cert

policy:
  minio-server:
    allow:
    - /v1/key/create/*
    - /v1/key/generate/*
    - /v1/key/decrypt/*
    - /v1/key/bulk/decrypt
    - /v1/key/list/*
    - /v1/status
    - /v1/metrics
    - /v1/log/audit
    - /v1/log/error
    identities:
    - ${MINIO_IDENTITY}

keys:
  - name: "minio-encryption-key-alpha"
  - name: "minio-encryption-key-baker"
  - name: "minio-encryption-key-charlie"

keystore:
  secretsmanager:
    endpoint: secretsmanager.REGION.amazonaws
    region: REGION
    kmskey: ""
    credentials:
      accesskey: "${AWS_ACCESS_KEY}"
      secretkey: "${AWS_SECRET_KEY}"

Reference

Key	Description
`address`	The network address and port the KES server listens to on startup. Defaults to port `7373` on all host network interfaces.
`root`	The identity for the KES superuser (`root`) identity. Clients connecting with a TLS certificate whose hash (`kes identity of client.cert`) matches this value have access to all KES API operations. Specify `disabled` to remove the root identity and rely only on the `policy` configuration for controlling identity and access management to KES.
`tls`	The TLS private key and certificate used by KES for establishing TLS-secured communications. Specify the full path for both the private `.key` and public `.cert` to the `key` and `cert` fields, respectively.
`policy`	Specify one or more policies to control access to the KES server. MinIO SSE requires access to the following KES cryptographic APIs: `/v1/key/create/` `/v1/key/generate/` `/v1/key/decrypt/` Specifying additional keys does not expand MinIO SSE functionality and may violate security best practices around providing unnecessary client access to cryptographic key operations. You can restrict the range of key names MinIO can create as part of performing SSE by specifying a prefix before the ``. For example, `minio-sse-*` only grants access to create, generate, or decrypt keys using the `minio-sse-` prefix. KES uses mTLS to authorize connecting clients by comparing the hash of the TLS certificate against the `identities` of each configured policy. Use the `kes identity of` command to compute the identity of the MinIO mTLS certificate and add it to the `policy.<NAME>.identities` array to associate MinIO to the `<NAME>` policy.
`keys`	Specify an array of keys which must exist on the root KMS for KES to successfully start. KES attempts to create the keys if they do not exist and exits with an error if it fails to create any key. KES does not accept any client requests until it completes validation of all specified keys.
`keystore.aws.secretsmanager`	The configuration for the AWS Secrets Manager and AWS KMS. `endpoint` - The endpoint for the Secrets Manager service, including the region. `approle` - The AWS region to use for other AWS services. `kmskey` - The root KMS Key to use for cryptographic operations. Formerly known as the Customer Master Key. `credentials` - The AWS Credentials to use for performing authenticated operations against Secrets Manager and KMS. The specified credentials must have the appropriate permissions