Server-Side Object Encryption with Hashicorp Vault Root KMS

Deploy or Ensure Access to a Hashicorp Vault Service

This procedure assumes an existing Hashicorp Vault installation accessible from the local host. The Vault Quick Start provides a sufficient foundation for the purposes of this procedure. Defer to the Vault Documentation for guidance on deployment and configuration.

MinIO KES supports either the V1 or V2 Vault K/V engines.

MinIO KES requires using AppRole authentication to the Vault server. You must create an AppRole, assign it a policy that the necessary permissions, and retrieve the AppRole ID and Secret for use in configuring KES.

You can use the following steps to enable AppRole authentication and create the necessary policies to support core KES functionality against Vault:

1. Enable AppRole Authentication

   vault auth enable approle
   

2. Create a Policy for KES

   Create a policy with necessary capabilities for KES to use when accessing Vault. Select the tab corresponding to the KV engine used for storing KES secrets:

   Vault Engine V1

   Create an access policy kes-policy.hcl with a configuration similar to the following:

   path "kv/*" {
         capabilities = [ "create", "read", "delete" ]
   }
   

   Write the policy to Vault using vault policy write kes-policy kes-policy.hcl.

   Vault Engine V2

   Create an access policy kes-policy.hcl with a configuration similar to the following:

   path "kv/data/*" {
         capabilities = [ "create", "read"]
   }
   
   path "kv/metadata/*" {
         capabilities = [ "list", "delete"]
   }
   

   Write the policy to Vault using vault policy write kes-policy kes-policy.hcl

3. Create an AppRole for KES and assign it the created policy

   vault write    auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m
   vault write    auth/approle/role/kes-role policies=kes-policy
   

4. Retrieve the AppRole ID and Secret

   vault read     auth/approle/role/kes-role/role-id
   vault write -f auth/approle/role/kes-role/secret-id
   

Deploy or Ensure Access to a MinIO Deployment

This procedure provides instructions for modifying the startup environment variables of a MinIO deployment to enable SSE via KES and the root KMS.

For instructions on new production deployments, see the Multi-Node Multi-Drive (Distributed) tutorial. For instructions on new local or evaluation deployments, see the Single-Node Single-Drive tutorial.

When creating the environment file for the deployment, pause and switch back to this tutorial to include the necessary environment variables to support SSE.

For existing MinIO Deployments, you can modify the existing environment file and restart the deployment as instructed during this procedure.

1) Download the KES Server Binary

Download the latest stable release (2023-05-02T22-48-10Z) of KES from github.com/minio/kes.

Select the binary appropriate for the host OS architecture. For example, hosts running X86-64 (Intel/AMD64) should download the kes-linux-amd64 package.

The following example code downloads the latest Linux AMD64-compatible binary and moves it to the system PATH:

curl --retry 10 https://github.com/minio/kes/releases/download/2023-05-02T22-48-10Z/kes-linux-amd64 -o /tmp/kes
chmod +x /tmp/kes
sudo mv /tmp/kes /usr/local/bin

kes --version

For distributed KES topologies, repeat this step and all following KES-specific instructions for each host on which you want to deploy KES. MinIO uses a round-robin approach by default for routing connections to multiple configured KES servers. For more granular controls, deploy a dedicated load balancer to manage connections to distributed KES hosts.

2) Generate TLS Certificates for KES and MinIO

The following commands create two TLS certificates that expire within 30 days of creation:

• A TLS certificate for KES to secure communications between it and the Vault deployment

• A TLS certificate for MinIO to perform mTLS authentication to KES.

# These commands output keys to /opt/kes/certs
# and /opt/minio/certs respectively

kes identity new kes_server \
  --key  /opt/kes/certs/kes-server.key  \
  --cert /opt/kes/certs/kes-server.cert  \
  --ip   "127.0.0.1"  \
  --dns  localhost

kes identity new minio_server \
  --key  /opt/minio/certs/minio-kes.key  \
  --cert /opt/minio/certs/minio-kes.cert  \
  --ip   "127.0.0.1"  \
  --dns  localhost

The --ip and --dns parameters set the IP and DNS SubjectAlternativeName for the certificate. The above example assumes that all components (Vault, MinIO, and KES) deploy on the same local host machine accessible via localhost or 127.0.0.1. You can specify additional IP or Hostnames based on the network configuration of your local host.

Depending on your Vault configuration, you may need to pass the kes-server.cert certificate as a trusted Certificate Authority. See the Hashicorp Server Configuration Documentation for more information. Defer to the client documentation for instructions on trusting a third-party CA.

3) Create the KES and MinIO Configurations

4) Start KES and MinIO

You must start KES before starting MinIO. The MinIO deployment requires access to KES as part of its startup.

Foreground processes depend on the shell or terminal in which they run. Exiting or terminating the shell/terminal instance also kills the attached process. Defer to your operating system best practices for running processes in the background.

5) Generate a New Encryption Key

MinIO requires that the EK exist on the root KMS before performing SSE operations using that key. Use kes key create or mc admin kms key create to add a new EK for use with SSE.

The following command uses the kes key create command to add a new External Key (EK) stored on the root KMS server for use with encrypting the MinIO backend.

export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=/opt/minio/certs/minio-kes.key
export KES_CLIENT_CERT=/opt/minio/certs/minio-kes.cert

kes key create -k encrypted-bucket-key

6) Enable SSE-KMS for a Bucket

You can use either the MinIO Console or the MinIO mc CLI to enable bucket-default SSE-KMS with the generated key:

mc alias set local http://127.0.0.1:9000 ROOTUSER ROOTPASSWORD

mc mb local/encryptedbucket
mc encrypt set SSE-KMS encrypted-bucket-key ALIAS/encryptedbucket

1) Download KES and Create the Service File

2) Generate TLS Certificates for KES and MinIO

Enabling connectivity between MinIO and KES requires at minimum one TLS certificate for performing mutual TLS (mTLS) authentication. Depending on your Vault configuration, you may also need to create a dedicated set of TLS certificates for KES to connect and authenticate to Vault. Defer to your organizations best practices around generating production-ready TLS certificates.

Place the certificates and corresponding private keys an appropriate directory such that the MinIO and KES service users can access and read their contents. The following example structure uses the folder hierarchy suggested in the beginning of this procedure:

-rw-r--r-- 1 kes:kes /opt/kes/certs/kes-server.cert
-rw-r--r-- 1 kes:kes /opt/kes/certs/kes-server.key

# If the Vault certs are self-signed or use a non-global CA
# Include those CA certs as well

-rw-r--r-- 1 kes:kes /opt/kes/certs/vault-CA.cert

-rw-r--r-- 1 minio-user:minio-user /opt/minio/certs/minio-kes.cert
-rw-r--r-- 1 minio-user:minio-user /opt/minio/certs/minio-kes.key

# If KES certs are self-signed or use a non-global CA
# Include the CA certs as well
-rw-r--r-- 1 minio-user:minio-user /opt/minio/certs/kes-server.cert

The general strategy for cert management is to ensure that each process (MinIO, KES, and Vault) have their own mTLS certificates and the Certificate Authority (CA) used to sign each client certificate.

Depending on your Vault configuration, you may also need to specify the CA used to sign the KES certificates to the Vault server. See the Hashicorp Vault Configuration Docs for more information. Defer to the client documentation for instructions on trusting a third-party CA.

3) Create the KES and MinIO Configurations

4) Start KES and MinIO

You must start KES before starting MinIO. The MinIO deployment requires access to KES as part of its startup.

This step uses systemd for starting and managing both the KES and MinIO server processes:

1. Start the KES Service on All Hosts

   Run the following command on each KES host to start the service:

   systemctl start kes
   

   You can validate the startup by using systemctl status kes. If the service started successfully, use journalctl -uf kes to check the KES output logs.

2. Start the MinIO Server

   For new MinIO deployments, run the following command on each MinIO host to start the service:

   systemctl start minio
   

   For existing MinIO deployments, run the following command on each MinIO host to restart the service:

   systemctl reload minio
   systemctl restart minio
   

5) Generate a New Encryption Key

MinIO requires that the EK exist on the root KMS before performing SSE operations using that key. Use kes key create or mc admin kms key create to add a new EK for use with SSE.

The following command uses the kes key create command to add a new External Key (EK) stored on the root KMS server for use with encrypting the MinIO backend.

export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=/opt/minio/certs/minio-kes.key
export KES_CLIENT_CERT=/opt/minio/certs/minio-kes.cert

kes key create -k encrypted-bucket-key

6) Enable SSE-KMS for a Bucket

You can use either the MinIO Console or the MinIO mc CLI to enable bucket-default SSE-KMS with the generated key:

mc alias set local http://127.0.0.1:9000 ROOTUSER ROOTPASSWORD

mc mb local/encryptedbucket
mc encrypt set SSE-KMS encrypted-bucket-key ALIAS/encryptedbucket