Hashicorp Vault Keystore

This tutorial shows how to setup a KES server that uses Vault’s K/V engine as a persistent and secure key store:

K E S C l i e n t K E S S e r v e r V a u l t

Vault Server Setup

  1. Generate Vault Private Key & Certificate

    KES and Vault exchange sensitive information. In particular, KES sends and receives the secret keys from Vault’s HTTP API. Therefore, it is necessary to secure the communication between KES and Vault.

    Here, we use self-signed certificates for simplicity.

    The following command generates a new TLS private key (vault.key) and a self-signed X.509 certificate (vault.crt) issued for the IP 127.0.0.1 and DNS name localhost:

    $ kes identity new --key vault.key --cert vault.crt --ip "127.0.0.1" localhost
    
      Private key:  vault.key
      Certificate:  vault.crt
      Identity:     37ced4538faa0c236b9fa80826b50de9afb45cc29acf6575f069a2d10e6125af
    
    If you already have a TLS private key & certificate, such as from WebPKI or an internal CA, you can use them instead. Remember to adjust the vault-config.json later on.
  2. Configure Vault Server

    The following vault-config.json starts a single Vault server instance on port 8200:

    {
      "api_addr": "https://127.0.0.1:8200",
      "backend": {
        "file": {
          "path": "vault/file"
        }
      },
    
      "default_lease_ttl": "168h",
      "max_lease_ttl": "720h",
    
      "listener": {
        "tcp": {
          "address": "0.0.0.0:8200",
          "tls_cert_file": "vault.crt",
          "tls_key_file": "vault.key",
          "tls_min_version": "tls12"
        }
      }
    }
    
    Note that we run Vault with a file backend. For high-availability you may want to use etcd, consul, or Vault with integrated storage instead.
  3. Start Vault Server

    Download the Vault binary.

    Linux Swap Protection:

    On linux, we can grant the binary the ipc_lock capability such that it can use the mlock syscall without root permissions:

    sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))
    

    Start the Vault server instance:

    $ vault server -config vault-config.json
    
  4. Set VAULT_ADDR endpoint

    The Vault CLI needs to know the Vault endpoint:

    export VAULT_ADDR='https://127.0.0.1:8200'
    
    Self-signed Certificates:

    When using a self-signed vault.crt the Vault CLI also needs to skip TLS certificate verification to talk to the Vault server:

    export VAULT_SKIP_VERIFY=true
    
  5. Initialize Vault Server

    $ vault operator init
    
    Unseal Key 1: eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
    Unseal Key 2: 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
    Unseal Key 3: cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF
    Unseal Key 4: /fTPpec5fWpGqWHK+uhnnTNMQyAbl5alUi4iq2yNgyqj
    Unseal Key 5: UPdDVPto+H6ko+20NKmagK40MOskqOBw4y/S51WpgVy/
    
    Initial Root Token: s.zaU4Gbcu0Wh46uj2V3VuUde0
    
    Vault is initialized with 5 key shares and a key threshold of 3. Please securely
    distribute the key shares printed above. When the Vault is re-sealed,
    restarted, or stopped, you must supply at least 3 of these keys to unseal it
    before it can start servicing requests.
    
    Vault prints N (5 by default) unseal key shares. Vault requires at least M (3 by default) unseal key shares to re-generate the actual unseal key to unseal Vault. Therefore, make sure to store them at a secure and durable location.
  6. Set VAULT_TOKEN

    The Vault CLI needs an authentication token to perform operations. The root access token is generated by vault operator init.

    export VAULT_TOKEN=s.zaU4Gbcu0Wh46uj2V3VuUde0
    

    Adjust the token to your own Vault access token.

  7. Unseal Vault Server

    Once initialized, unseal the Vault using M out of N unseal key shares:

    vault operator unseal eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
    
    vault operator unseal 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
    
    vault operator unseal cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF
    

    After submitting enough valid unseal key shares, Vault unseals and can process requests.

  8. Enable K/V Backend

    KES stores the secret keys at the Vault K/V backend. Vault provides two K/V engines, v1 and v2.

    MinIO recommends the K/V v1 engine.

    The following command enables the K/V v1 secret engine:

    vault secrets enable -version=1 kv
    

    The following command enables the K/V v2 secret engine:

    $ vault secrets enable -version=2 kv
    
    Note that the Vault policy for KES depends on the chosen K/V engine version. The v2 engine requires slightly different policy rules compared to the v1 engine. For more information about migrating from v1 to v2 see upgrading from v1.
  9. Create Vault Policy

    The Vault policy defines the API paths the KES server can access.

    • For v1

      The following kes-policy.hcl policy should be used for the K/V v1 backend:

      path "kv/*" {
         capabilities = [ "create", "read", "delete" ]
      }
      
    • For v2 T The following kes-policy.hcl policy should be used for the K/V v2 backend:

      path "kv/data/*" {
         capabilities = [ "create", "read" ]
      }
      path "kv/metadata/*" {
         capabilities = [ "list", "delete" ]       
      }
      

    The following command creates the policy at Vault:

    vault policy write kes-policy kes-policy.hcl
    
  10. Enable AppRole Authentication

    This step allows the KES server to authenticate to Vault. For this tutorial, we use the AppRole authentication method.

    vault auth enable approle
    
  11. Create KES Role

    The following command adds a new role kes-server at Vault:

    vault write auth/approle/role/kes-server token_num_uses=0  secret_id_num_uses=0  period=5m
    
  12. Bind Policy to Role

    The following command binds kes-server role to the key-policy:

    vault write auth/approle/role/kes-server policies=kes-policy
    
  13. Generate AppRole ID

    Request an AppRole ID for the KES server:

    vault read auth/approle/role/kes-server/role-id 
    
  14. Generate AppRole Secret

    Request an AppRole secret for the KES server:

    vault write -f auth/approle/role/kes-server/secret-id 
    

    The AppRole secret prints as secret_id. You can ignore the secret_id_accessor.

KES Server Setup

  1. Generate KES Server Private Key & Certificate

    The following command generates a new TLS private key server.key and a self-signed X.509 certificate server.cert that is issued for the IP 127.0.0.1 and DNS name localhost (as SAN). Customize the command to match your setup.

    kes tool identity new --server --key server.key --cert server.cert --ip "127.0.0.1" --dns localhost
    

    Any other tooling for X.509 certificate generation works as well. For example, you could use openssl:

    $ openssl ecparam -genkey -name prime256v1 | openssl ec -out server.key
    
    $ openssl req -new -x509 -days 30 -key server.key -out server.cert \
        -subj "/C=/ST=/L=/O=/CN=localhost" -addext "subjectAltName = IP:127.0.0.1"
    
  2. Generate Client Credentials

    The following command generates a new TLS private/public key pair for the client application to use for the KES Server:

    $ kes identity new --key=client.key --cert=client.crt MyApp
    
      Private key:  client.key
      Certificate:  client.crt
      Identity:     02ef5321ca409dbc7b10e7e8ee44d1c3b91e4bf6e2198befdebee6312745267b
    

    The identity 02ef5321ca409dbc7b10e7e8ee44d1c3b91e4bf6e2198befdebee6312745267b is a unique fingerprint of the public key in client.crt. You can re-compute it anytime:

    $ kes identity of client.crt
    
      Identity:  02ef5321ca409dbc7b10e7e8ee44d1c3b91e4bf6e2198befdebee6312745267b
    
  3. Configure KES Server

    Create the KES server configuration file: config.yml.

    Make sure that the identity in the policy section matches the client.crt identity. Add the approle role_id and secret_id obtained earlier.

    address: 0.0.0.0:7373 # Listen on all network interfaces on port 7373
    
    admin:
      identity: disabled  # We disable the admin identity since we don't need it in this guide 
    
    tls:
      key: private.key    # The KES server TLS private key
      cert: public.crt    # The KES server TLS certificate
    
    policy:
      my-app: 
        allow:
        - /v1/key/create/my-key*
        - /v1/key/generate/my-key*
        - /v1/key/decrypt/my-key*
        identities:
        - 02ef5321ca409dbc7b10e7e8ee44d1c3b91e4bf6e2198befdebee6312745267b # Use the identity of your client.crt
    
    keystore:
       vault:
         endpoint: https://127.0.0.1:8200
         version:  v1 # The K/V engine version - either "v1" or "v2".
         approle:
           id:     "" # Your AppRole ID
           secret: "" # Your AppRole Secret
           retry:  15s
         status:
           ping: 10s
         tls:
           ca: vault.crt # Manually trust the vault certificate since we use self-signed certificates
    
  4. Start KES Server

    Linux

    kes server --config config.yml --auth off
    
    Linux Swap Protection:

    In Linux environments, KES can use the mlock syscall to prevent the OS from writing in-memory data to disk (swapping). This prevents leaking sensitive data.

    Use the following command to allow KES to use the mlock syscall without running with root privileges:

    $ sudo setcap cap_ipc_lock=+ep $(readlink -f $(which kes))
    

    Start a KES server instance with memory protection:

    kes server --config config.yml --auth off --mlock
    

    Containers

    The following instructions use Podman to manage the containers. You can also use Docker.

    Modify addresses and file paths as needed for your deployment.

    sudo podman pod create  \
      -p 9000:9000 -p 9001:9001 -p 7373:7373  \
      -v ~/minio-kes-vault/certs:/certs  \
      -v ~/minio-kes-vault/minio:/mnt/minio  \
      -v ~/minio-kes-vault/config:/etc/default/  \
      -n minio-kes-vault
    
    sudo podman run -dt  \
      --cap-add IPC_LOCK  \
      --name kes-server  \
      --pod "minio-kes-vault"  \
      -e KES_SERVER=https://127.0.0.1:7373  \
      -e KES_CLIENT_KEY=/certs/kes-server.key  \
      -e KES_CLIENT_CERT=/certs/kes-server.cert  \
      quay.io/minio/kes:2024-01-11T13-09-29Z server  \
        --auth  \
        --config=/etc/default/kes-config.yaml  \
    
    sudo podman run -dt  \
      --name minio-server  \
      --pod "minio-kes-vault"  \
      -e "MINIO_CONFIG_ENV_FILE=/etc/default/minio"  \
      quay.io/minio/minio:RELEASE.2024-01-31T20-20-33Z server  \
        --console-address ":9001"
    

    You can verify the status of the containers using the following command. The command should show three pods, one for the Pod, one for KES, and one for MinIO.

    sudo podman container list
    

KES CLI Access

  1. Set KES_SERVER Endpoint

    The following environment variable specifies the server the KES CLI should talk to:

    export KES_SERVER=https://127.0.0.1:7373
    
  2. Define the Client Credentials

    The following environment variables set the access credentials the client uses to talk to a KES server:

    export KES_CLIENT_CERT=client.crt
    
    export KES_CLIENT_KEY=client.key
    
  3. Test the Configuration

    Perform any API operation allowed by the policy we assigned above.

    For example, create a key:

    kes key create my-key-1
    

    Use the key to generate a new data encryption key:

    $ kes key dek my-key-1
    {
      plaintext : UGgcVBgyQYwxKzve7UJNV5x8aTiPJFoR+s828reNjh0=
      ciphertext: eyJhZWFkIjoiQUVTLTI1Ni1HQ00tSE1BQy1TSEEtMjU2IiwiaWQiOiIxMTc1ZjJjNDMyMjNjNjNmNjY1MDk5ZDExNmU3Yzc4NCIsIml2IjoiVHBtbHpWTDh5a2t4VVREV1RSTU5Tdz09Iiwibm9uY2UiOiJkeGl0R3A3bFB6S21rTE5HIiwiYnl0ZXMiOiJaaWdobEZrTUFuVVBWSG0wZDhSYUNBY3pnRWRsQzJqWFhCK1YxaWl2MXdnYjhBRytuTWx0Y3BGK0RtV1VoNkZaIn0=
    }
    

    To run KES locally for testing purposes, use the -k or -insecure flag to generate a new data encryption key:

    $ kes key dek my-key-1 -k
    {
      plaintext : UGgcVBgyQYwxKzve7UJNV5x8aTiPJFoR+s828reNjh0=
      ciphertext: eyJhZWFkIjoiQUVTLTI1Ni1HQ00tSE1BQy1TSEEtMjU2IiwiaWQiOiIxMTc1ZjJjNDMyMjNjNjNmNjY1MDk5ZDExNmU3Yzc4NCIsIml2IjoiVHBtbHpWTDh5a2t4VVREV1RSTU5Tdz09Iiwibm9uY2UiOiJkeGl0R3A3bFB6S21rTE5HIiwiYnl0ZXMiOiJaaWdobEZrTUFuVVBWSG0wZDhSYUNBY3pnRWRsQzJqWFhCK1YxaWl2MXdnYjhBRytuTWx0Y3BGK0RtV1VoNkZaIn0=
    }
    

Using KES with a MinIO Server

MinIO Server requires KES to enable server-side data encryption.

See the KES for MinIO instruction guide for additional steps needed to use your new KES Server with a MinIO Server.

Configuration References

The following section describes the Key Encryption Service (KES) configuration settings to use Hashicorp Vault Keystore as the root KMS to store external keys, such as the keys used for Server-Side Encryption on a MinIO Server.

MinIO Server Requires Expanded Permissions:
Starting with MinIO Server RELEASE.2023-02-17T17-52-43Z, MinIO requires expanded KES permissions for functionality. The example configuration in this section contains all required permissions.

Advanced Configuration

These additional configuration steps may solve specific problems.

Multi-Tenancy with K/V prefixes

Vault can serve as backend for multiple, isolated KES tenants. Each KES tenant can consist of N replicas. There can be M KES tenants connected to the same Vault server/cluster.

This means N × M KES server instances can connect to a single Vault.

In these configurations, each KES tenant has a separate prefix at the K/V secret engine. For each KES tenant, there must be a corresponding Vault policy.

  • For K/V v1:

    path "kv/<tenant-name>/*" {
       capabilities = [ "create", "read", "delete" ]
    }
    
  • For K/V v2:

    path "kv/data/<tenant-name>/*" {
      capabilities = [ "create", "read" ]
    }
    path "kv/metadata/<tenant-name>/*" {
      capabilities = [ "list", "delete" ]       
    }
    

Create a different configuration file for each KES tenant. The file contains the Vault K/V prefix for the tenant to use.

keystore:
   vault:
     endpoint: https://127.0.0.1:8200
     prefix: <tenant-name>
     approle:
       id:     "" # Your AppRole ID
       secret: "" # Your AppRole Secret
       retry:  15s
     status:
       ping: 10s
     tls:
       ca: vault.crt # Manually trust the vault certificate since we use self-signed certificates

Multi-Tenancy with Vault Namespaces

Vault can serve as the backend for multiple, isolated KES tenants. Each KES tenant can consist of N replicas. There can be M KES tenants connected to the same Vault server/cluster.

This means N × M KES server instances can connect to a single Vault.

Therefore, each KES tenant has a separate prefix at the K/V secret engine. For each KES tenant there has to be a corresponding Vault policy.

  • For K/V v1:

    path "kv/<tenant-name>/*" {
       capabilities = [ "create", "read", "delete" ]
    }
    
  • For K/V v2:

    path "kv/data/<tenant-name>/*" {
       capabilities = [ "create", "read" ]
    }
    path "kv/metadata/<tenant-name>/*" {
       capabilities = [ "list", "delete" ]       
    }
    

Use a different configuration file for each KES tenant. The file contains the Vault namespace which the KES tenant should use.

keystore:
   vault:
     endpoint: https://127.0.0.1:8200
     namespace: <vault-namespace>
     approle:
       id:     "" # Your AppRole ID
       secret: "" # Your AppRole Secret
       retry:  15s
     status:
       ping: 10s
     tls:
       ca: vault.crt # Manually trust the vault certificate since we use self-signed certificates

Encrypt Vault-stored Keys

Hashicorp’s Transit functionality provides a means to encrypt and decrypt keys stored in the vault. This provides an additional layer of encryption that may be useful in specific use cases.

When enabled, Hashicorp stores a key in the Vault to encrypt or decrypt the other keys stored in the vault. KES then uses the vault-managed key to store or retrieve keys from the Vault.

If the specified transit key is incorrect, disabled, removed, or otherwise unaccessible, KES cannot retrieve any vault keys nor perform any en/decryption operations relying on those keys.

To configure Transit, add the following section to the KES Configuration YAML’s keystore.vault section:

keystore:
  vault:
    transit:      # Optionally encrypt keys stored on the K/V engine with a Vault-managed key.
      engine: ""  # The path of the transit engine - e.g. "my-transit". If empty, defaults to: transit (Vault default)
      key: ""     # The key name that should be used to encrypt entries stored on the K/V engine.

References