Documentation

Server-Side Object Encryption with Hashicorp Vault Root KMS

This procedure provides guidance for deploying MinIO configured to use KES and enable Server Side Encryption.

As part of this procedure, you will:

  1. Deploy one or more KES servers configured to use Hashicorp Vault as the root KMS. You may optionally deploy a load balancer for managing connections to those KES servers.

  2. Create a new EK on Vault for use with SSE.

  3. Create or modify a MinIO deployment with support for SSE using KES. Defer to the Deploy Distributed MinIO tutorial for guidance on production-ready MinIO deployments.

  4. Configure automatic bucket-default SSE-KMS

Important

Enabling SSE on a MinIO deployment automatically encrypts the backend data for that deployment using the default encryption key.

MinIO requires access to KES and the root KMS to decrypt the backend and start normally. You cannot disable KES later or “undo” the SSE configuration at a later point.

Prerequisites

Deploy or Ensure Access to a Hashicorp Vault Service

This procedure assumes an existing Hashicorp Vault installation accessible from the local host. The Vault Quick Start provides a sufficient foundation for the purposes of this procedure. Defer to the Vault Documentation for guidance on deployment and configuration.

KES Operations Require Unsealed Vault

You must unseal the Vault instance to allow any cryptographic operations, including key creation and retrieval. KES returns an error if the configured Vault service is sealed.

If you restart or otherwise seal the Vault instance, KES cannot perform any cryptographic operations against the Vault. You must unseal the Vault to ensure normal operations.

See the Vault documentation on Seal/Unseal for more information.

MinIO KES supports either the V1 or V2 Vault K/V engines.

MinIO KES requires using AppRole authentication to the Vault server. You must create an AppRole, assign it a policy that the necessary permissions, and retrieve the AppRole ID and Secret for use in configuring KES.

You can use the following steps to enable AppRole authentication and create the necessary policies to support core KES functionality against Vault:

  1. Enable AppRole Authentication

    vault auth enable approle
    
  2. Create a Policy for KES

    Create a policy with necessary capabilities for KES to use when accessing Vault. Select the tab corresponding to the KV engine used for storing KES secrets:

    Create an access policy kes-policy.hcl with a configuration similar to the following:

    path "kv/*" {
          capabilities = [ "create", "read", "delete" ]
    }
    

    Write the policy to Vault using vault policy write kes-policy kes-policy.hcl.

    Create an access policy kes-policy.hcl with a configuration similar to the following:

    path "kv/data/*" {
          capabilities = [ "create", "read"]
    }
    
    path "kv/metadata/*" {
          capabilities = [ "list", "delete"]
    }
    

    Write the policy to Vault using vault policy write kes-policy kes-policy.hcl

  3. Create an AppRole for KES and assign it the created policy

    vault write    auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m
    vault write    auth/approle/role/kes-role policies=kes-policy
    
  4. Retrieve the AppRole ID and Secret

    vault read     auth/approle/role/kes-role/role-id
    vault write -f auth/approle/role/kes-role/secret-id
    

Deploy or Ensure Access to a MinIO Deployment

This procedure provides instructions for modifying the startup environment variables of a MinIO deployment to enable SSE via KES and the root KMS.

For instructions on new production deployments, see the Multi-Node Multi-Drive (Distributed) tutorial. For instructions on new local or evaluation deployments, see the Single-Node Single-Drive tutorial.

When creating the environment file for the deployment, pause and switch back to this tutorial to include the necessary environment variables to support SSE.

For existing MinIO Deployments, you can modify the existing environment file and restart the deployment as instructed during this procedure.

Procedure

This procedure provides instructions for configuring and enabling Server-Side Encryption using Hashicorp Vault in production environments. Specifically, this procedure assumes the following:

  • An existing production-grade Vault deployment

  • One or more hosts for deploying KES

  • One or more hosts for a new or existing MinIO deployment

1) Download KES and Create the Service File

  1. Download KES

    Download the latest stable release (2023-11-10T10-44-28Z) of KES from github.com/minio/kes.

    Select the binary appropriate for the host OS architecture. For example, hosts running X86-64 (Intel/AMD64) should download the kes-linux-amd64 package.

    The following example code downloads the latest Linux AMD64-compatible binary and moves it to the system PATH:

    curl --retry 10 https://github.com/minio/kes/releases/download/2023-11-10T10-44-28Z/kes-linux-amd64 -o /tmp/kes
    chmod +x /tmp/kes
    sudo mv /tmp/kes /usr/local/bin
    
    kes --version
    

    For distributed KES topologies, repeat this step and all following KES-specific instructions for each host on which you want to deploy KES. MinIO uses a round-robin approach by default for routing connections to multiple configured KES servers. For more granular controls, deploy a dedicated load balancer to manage connections to distributed KES hosts.

  2. Create the Service File

    Create the /lib/systemd/system/kes.service file on all KES hosts:

    [Unit]
    Description=KES
    Documentation=https://github.com/minio/kes/wiki
    Wants=network-online.target
    After=network-online.target
    AssertFileIsExecutable=/usr/local/bin/kes
    
    [Service]
    WorkingDirectory=/etc/kes/
    
    AmbientCapabilities=CAP_IPC_LOCK
    
    User=kes
    Group=kes
    ProtectProc=invisible
    
    ExecStart=/usr/local/bin/kes server --config=/opt/kes/config.yaml
    
    # Let systemd restart this service always
    Restart=always
    
    # Specifies the maximum file descriptor number that can be opened by this process
    LimitNOFILE=65536
    
    # Specifies the maximum number of threads this process can create
    TasksMax=infinity
    
    # Disable timeout logic and wait until process is stopped
    TimeoutStopSec=infinity
    SendSIGKILL=no
    
    [Install]
    WantedBy=multi-user.target
    

    You may need to run systemctl daemon-reload to load the new service file into systemctl.

    The kes.service file runs as the kes User and Group by default. You can create the user and group using the useradd and groupadd commands. The following example creates the user and group. These commands typically require root (sudo) permissions.

    groupadd -r kes
    useradd -M -r -g kes kes
    

    The kes user and group must have read access to all files used by the KES service:

    chown -R kes:kes /opt/kes
    

2) Generate TLS Certificates for KES

KES requires TLS connectivity for all client connections, including those originating from MinIO. See Network Encryption (TLS) for more information on enabling TLS for the MinIO deployment.

Depending on your Vault configuration, you may also need to create a dedicated set of TLS certificates for KES to connect and authenticate to Vault.

Defer to your organization’s best practices around generating production-ready TLS certificates.

Place the certificates and corresponding private keys in a directory that the KES service user has permissions to access and read the directory’s contents. For example:

-rw-r--r-- 1 kes:kes /opt/kes/certs/kes-server.cert
-rw-r--r-- 1 kes:kes /opt/kes/certs/kes-server.key

# If the Vault certs are self-signed or use a non-global CA
# Include those CA certs as well

-rw-r--r-- 1 kes:kes /opt/kes/certs/vault-CA.cert

Depending on your Vault configuration, you may also need to specify the CA used to sign the KES certificates to the Vault server. See the Hashicorp Vault Configuration Docs for more information. Defer to the client documentation for instructions on trusting a third-party CA.

3) Generate a KES API Key for use by MinIO

Starting with KES version 2023-02-15T14-54-37Z, you can generate an API key to use for authenticating to the KES server.

Use the kes identity new command to generate a new API key for use by the MinIO Server:

kes identity new

The output includes both the API Key for use with MinIO and the Identity hash for use with the KES Policy configuration.

4) Create the KES and MinIO Configurations

Important

Starting with https://github.com/minio/minio/releases/tag/RELEASE.2023-02-17T17-52-43Z, MinIO requires expanded KES permissions for functionality. The example configuration in this section contains all required permissions.

  1. Create the KES Configuration File

    Create the configuration file using your preferred text editor. The following example uses nano:

    nano /opt/kes/config.yaml
    

    KES uses a YAML-formatted configuration file. The following YAML provides the minimum required fields for using Hashicorp Vault as the root KMS. You must modify this YAML to reflect your deployment environment.

    address: 0.0.0.0:7373
    
    # Disable the root administrator identity, as we do not need that level of access for
    # supporting SSE operations.
    admin:
      identity: disabled
    
    # Specify the TLS keys generated in the previous step here
    # For production environments, use keys signed by a known and trusted Certificate Authority (CA).
    tls:
      key:  /opt/kes/certs/kes-server.key
      cert: /opt/kes/certs/kes-server.cert
    
      # Specify the path to CAs used by KES for validating client certificates
      # This can alternatively be a single CA
      # KES uses these CAs in addition to the system trust store for validating client certificates.
    
      ca: /opt/kes/certs/CAs/
    
    # Sets access policies for KES
    # The `minio` policy grants access to the listed APIs.
    policy:
      minio:
        allow:
        - /v1/key/create/*   # You can replace these wildcard '*' with a string prefix to restrict key names
        - /v1/key/generate/* # e.g. '/minio-'
        - /v1/key/decrypt/*
        - /v1/key/bulk/decrypt
        - /v1/key/list/*
        - /v1/status
        - /v1/metrics
        - /v1/log/audit
        - /v1/log/error
        identities:
        - MINIO_API_KEY_HASH # Replace with the hash output returned from kes identity new
    
    # Specify the connection information for the Vault server.
    # The endpoint should be resolvable from the host.
    # This example assumes that Vault is configured with an AppRole ID and
    # Secret for use with KES.
    keystore:
      vault:
        endpoint: https://HOSTNAME:8200
        engine: "/path/to/engine" # Replace with the path to the K/V Engine
        version: "v1|v2" # Specify v1 or v2 depending on the version of the K/V Engine
        approle:
          id: "VAULTAPPID"     # Hashicorp Vault AppRole ID
          secret: "VAULTAPPSECRET" # Hashicorp Vault AppRole Secret ID
          retry: 15s
        status:
          ping: 10s
        # Required if Vault uses certificates signed by an unknown CA,
        # e.g. self-signed or internal (non-globally trusted).
        # Replace this value with the full path to the Vault CA certificate.
        tls:
          ca: vault-tls-CA.cert
    
    • Set MINIO_IDENTITY_HASH to the identity hash of the API Key generated in the previous step.

      The following command recomputes the necessary hash from the API key:

      kes identity of kes:v1:KEY/KEY
      
    • Replace the keystore.vault.endpoint with the hostname of the Vault server(s).

    • Replace keystore.vault.engine and keystore.vault.version with the path and version of the KV engine used for storing secrets.

    • Replace the VAULTAPPID and VAULTAPPSECRET with the appropriate Vault AppRole credentials.

    • Modify the keystore.vault.tls.ca value to correspond to the path to the Vault CA certificate used to sign the Vault TLS keys.

  2. Configure the MinIO Environment File

    Create or modify the MinIO Server environment file for all hosts in the target deployment to include the following environment variables:

    Add the following lines to the MinIO Environment file on each MinIO host. See the tutorials for Deploy MinIO: Single-Node Single-Drive, Deploy MinIO: Single-Node Multi-Drive, or Deploy MinIO: Multi-Node Multi-Drive for more detailed descriptions of a base MinIO environment file.

    # Add these environment variables to the existing environment file
    
    MINIO_KMS_KES_ENDPOINT=https://HOSTNAME:7373
    MINIO_KMS_KES_API_KEY="kes:v1:ACTpAsNoaGf2Ow9o5gU8OmcaG6Af/VcZ1Mt7ysuKoBjv"
    
    # Allows validation of the KES Server Certificate (Self-Signed or Third-Party CA)
    # Change this path to the location of the KES CA Path
    MINIO_KMS_KES_CAPATH=/opt/kes/certs/kes-server.cert
    
    # Sets the default KMS key for the backend and SSE-KMS/SSE-S3 Operations)
    MINIO_KMS_KES_KEY_NAME=minio-backend-default-key
    
    # Optional, defines the name for the KES server enclave to use.
    # MINIO_KMS_KES_ENCLAVE=<name>
    

    Replace HOSTNAME with the IP address or hostname of the KES server. If the MinIO server host machines cannot resolve or reach the specified HOSTNAME, the deployment may return errors or fail to start.

    • If using a single KES server host, specify the IP or hostname of that host

    • If using multiple KES server hosts, specify a comma-separated list of IPs or hostnames of each host

    MinIO uses the MINIO_KMS_KES_KEY_NAME key for the following cryptographic operations:

    • Encrypting the MinIO backend (IAM, configuration, etc.)

    • Encrypting objects using SSE-KMS if the request does not include a specific EK.

    • Encrypting objects using SSE-S3.

    MinIO uses the MINIO_KMS_KES_ENCLAVE key to define the name of the KES enclave to use.

    • Replace <name> with the name of the enclave to use. If not defined, MinIO does not send any enclave information. This may result in using the default enclave for stateful KES servers.

      A KES enclave provides an isolated space for its associated keys separate from other enclaves on a stateful KES server.

    MinIO defaults to expecting this file at /etc/default/minio. If you modified your deployment to use a different location for the environment file, modify the file at that location.

5) Start KES and MinIO

KES Operations Requires Unsealed Vault

You must unseal the Vault instance to allow normal cryptographic operations, including key creation or retrieval. See the Vault documentation on Seal/Unseal for more information.

You must start KES before starting MinIO. The MinIO deployment requires access to KES as part of its startup.

This step uses systemd for starting and managing both the KES and MinIO server processes:

  1. Start the KES Service on All Hosts

    Run the following command on each KES host to start the service:

    systemctl start kes
    

    You can validate the startup by using systemctl status kes. If the service started successfully, use journalctl -uf kes to check the KES output logs.

  2. Start the MinIO Server

    For new MinIO deployments, run the following command on each MinIO host to start the service:

    systemctl start minio
    

    For existing MinIO deployments, run the following command on each MinIO host to restart the service:

    systemctl reload minio
    systemctl restart minio
    

5) Generate a New Encryption Key

KES Operations Requires Unsealed Vault

You must unseal the Vault instance to allow normal cryptographic operations, including key creation or retrieval. See the Vault documentation on Seal/Unseal for more information.

MinIO requires that the EK exist on the root KMS before performing SSE operations using that key. Use kes key create or mc admin kms key create to add a new EK for use with SSE.

The following command uses the mc admin kms key create command to add a new External Key (EK) stored on the root KMS server for use with encrypting the MinIO backend.

mc admin kms key create ALIAS KEYNAME

6) Enable SSE-KMS for a Bucket

You can use either the MinIO Console or the MinIO mc CLI to enable bucket-default SSE-KMS with the generated key:

Open the MinIO Console by navigating to http://127.0.0.1:9090 in your preferred browser and logging in with the root credentials specified to the MinIO container. If you deployed MinIO using a different Console listen port, substitute 9090 with that port value.

Once logged in, create a new Bucket and name it to your preference. Select the Gear icon to open the management view.

Select the pencil icon next to the Encryption field to open the modal for configuring a bucket default SSE scheme.

Select SSE-KMS, then enter the name of the key created in the previous step.

Once you save your changes, try to upload a file to the bucket. When viewing that file in the object browser, note that in the sidebar the metadata includes the SSE encryption scheme and information on the key used to encrypt that object. This indicates the successful encrypted state of the object.

The following commands:

  • Create a new alias for the MinIO deployment

  • Create a new bucket for storing encrypted data

  • Enable SSE-KMS encryption on that bucket

mc alias set local http://127.0.0.1:9000 ROOTUSER ROOTPASSWORD

mc mb local/encryptedbucket
mc encrypt set SSE-KMS encrypted-bucket-key ALIAS/encryptedbucket

Write a file to the bucket using mc cp or any S3-compatible SDK with a PutObject function. You can then run mc stat on the file to confirm the associated encryption metadata.

Configuration Reference for Hashicorp Vault

The following section describes each of the Key Encryption Service (KES) configuration settings for using Hashicorp Vault as the root Key Management Service (KMS) for SSE.

Important

Starting with https://github.com/minio/minio/releases/tag/RELEASE.2023-02-17T17-52-43Z, MinIO requires expanded KES permissions for functionality. The example configuration in this section contains all required permissions.

The following YAML describes the minimum required fields for configuring Hashicorp Vault as an external KMS for supporting SSE.

Fields with ${<STRING>} use the environment variable matching the <STRING> value. You can use this functionality to set credentials without writing them to the configuration file.

The YAML assumes a minimal set of permissions for the MinIO deployment accessing KES. As an alternative, you can omit the policy.minio-server section and instead set the ${MINIO_IDENTITY} hash as the ${ROOT_IDENTITY}.

address: 0.0.0.0:7373
admin:
  identity: ${ROOT_IDENTITY}

tls:
  key: kes-server.key
  cert: kes-server.cert

policy:
  minio-server:
    allow:
    - /v1/key/create/*
    - /v1/key/generate/*
    - /v1/key/decrypt/*
    - /v1/key/bulk/decrypt
    - /v1/key/list/*
    - /v1/status
    - /v1/metrics
    - /v1/log/audit
    - /v1/log/error
    identities:
    - ${MINIO_IDENTITY}

keys:
  - name: "minio-encryption-key-alpha"
  - name: "minio-encryption-key-baker"
  - name: "minio-encryption-key-charlie"

keystore:
  vault:
    endpoint: https://vault.example.net:8200
    engine: "kv"
    version: "v1"
    namespace: "minio"
    prefix: "keys"
    approle:
      id: ${KES_APPROLE_ID}
      secret: ${KES_APPROLE_SECRET}
      retry: 15s
    status:
      ping: 10s
    tls:
      key: "kes-mtls.key"
      cert: "kes-mtls.cert"
      ca: vault-tls.cert

Key

Description

address

The network address and port the KES server listens to on startup. Defaults to port 7373 on all host network interfaces.

root

The identity for the KES superuser (root) identity. Clients connecting with a TLS certificate whose hash (kes identity of client.cert) matches this value have access to all KES API operations.

Specify disabled to remove the root identity and rely only on the policy configuration for controlling identity and access management to KES.

tls

The TLS private key and certificate used by KES for establishing TLS-secured communications. Specify the full path for both the private .key and public .cert to the key and cert fields, respectively.

policy

Specify one or more policies to control access to the KES server.

MinIO SSE requires access to the following KES cryptographic APIs:

  • /v1/key/create/*

  • /v1/key/generate/*

  • /v1/key/decrypt/*

Specifying additional keys does not expand MinIO SSE functionality and may violate security best practices around providing unnecessary client access to cryptographic key operations.

You can restrict the range of key names MinIO can create as part of performing SSE by specifying a prefix before the *. For example, minio-sse-* only grants access to create, generate, or decrypt keys using the minio-sse- prefix.

KES uses mTLS to authorize connecting clients by comparing the hash of the TLS certificate against the identities of each configured policy. Use the kes identity of command to compute the identity of the MinIO mTLS certificate and add it to the policy.<NAME>.identities array to associate MinIO to the <NAME> policy.

keys

Specify an array of keys which must exist on the root KMS for KES to successfully start. KES attempts to create the keys if they do not exist and exits with an error if it fails to create any key. KES does not accept any client requests until it completes validation of all specified keys.

keystore.vault

The configuration for the Hashicorp Vault keystore. The following fields are required:

  • endpoint - The hostname for the vault server(s). The hostname must be resolvable by the KES server host.

  • engine - The path to the K/V engine to use. Defaults to kv

  • version - The version of the K/V engine to use.

    Specify either v1 or v2. Defaults to v1.

  • namespace - The Vault namespace to use for secret storage.

  • prefix - The prefix to use for secret storage.

  • approle - The AppRole used by KES for performing authenticated operations against Vault.

    The specified AppRole must have the appropriate permissions

  • tls.ca - The Certificate Authority used to sign the Vault TLS certificates. Typically required if the Vault server uses self-signed certificates or is signed by an unknown CA (internal or non-global).