KES Documentation
Customer Login

Documentation

Hashicorp Vault Keystore

This tutorial shows how to setup a KES server that uses Vault’s K/V engine as a persistent and secure key store:

K E S C l i e n t K E S S e r v e r V a u l t

Vault Server Setup

  1. Generate Vault Private Key & Certificate

    KES and Vault exchange sensitive information. In particular, KES sends and receives the secret keys from Vault’s HTTP API. Therefore, it is necessary to secure the communication between KES and Vault.

    Here, we use self-signed certificates for simplicity.

    The following command generates a new TLS private key (vault.key) and a self-signed X.509 certificate (vault.crt) issued for the IP 127.0.0.1 and DNS name localhost:

    $ kes identity new --key vault.key --cert vault.crt --ip "127.0.0.1" localhost

  Private key:  vault.key
  Certificate:  vault.crt
  Identity:     37ced4538faa0c236b9fa80826b50de9afb45cc29acf6575f069a2d10e6125af
    If you already have a TLS private key & certificate, such as from WebPKI or an internal CA, you can use them instead. Remember to adjust the vault-config.json later on.

  2. Configure Vault Server

    The following vault-config.json starts a single Vault server instance on port 8200:

    {
  "api_addr": "https://127.0.0.1:8200",
  "backend": {
    "file": {
      "path": "vault/file"
    }
  },

  "default_lease_ttl": "168h",
  "max_lease_ttl": "720h",

  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_cert_file": "vault.crt",
      "tls_key_file": "vault.key",
      "tls_min_version": "tls12"
    }
  }
}
    Note that we run Vault with a file backend. For high-availability you may want to use etcd, consul, or Vault with integrated storage instead.

  3. Start Vault Server

    Download the Vault binary.

    Linux Swap Protection:

    On linux, we can grant the binary the ipc_lock capability such that it can use the mlock syscall without root permissions:

    sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))

    Start the Vault server instance:

    $ vault server -config vault-config.json

  4. Set VAULT_ADDR endpoint

    The Vault CLI needs to know the Vault endpoint:

    export VAULT_ADDR='https://127.0.0.1:8200'
    Self-signed Certificates:

    When using a self-signed vault.crt the Vault CLI also needs to skip TLS certificate verification to talk to the Vault server:

    export VAULT_SKIP_VERIFY=true

  5. Initialize Vault Server

    $ vault operator init

Unseal Key 1: eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
Unseal Key 2: 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
Unseal Key 3: cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF
Unseal Key 4: /fTPpec5fWpGqWHK+uhnnTNMQyAbl5alUi4iq2yNgyqj
Unseal Key 5: UPdDVPto+H6ko+20NKmagK40MOskqOBw4y/S51WpgVy/

Initial Root Token: s.zaU4Gbcu0Wh46uj2V3VuUde0

Vault is initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
    Vault prints N (5 by default) unseal key shares. Vault requires at least M (3 by default) unseal key shares to re-generate the actual unseal key to unseal Vault. Therefore, make sure to store them at a secure and durable location.

  6. Set VAULT_TOKEN

    The Vault CLI needs an authentication token to perform operations. The root access token is generated by vault operator init.

    $ export VAULT_TOKEN=s.zaU4Gbcu0Wh46uj2V3VuUde0

    Adjust the token to your own Vault access token.

  7. Unseal Vault Server

    Once initialized, unseal the Vault using M out of N unseal key shares:

    $ vault operator unseal eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT

    $ vault operator unseal 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2

    $ vault operator unseal cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF

    After submitting enough valid unseal key shares, Vault unseals and can process requests.

  8. Enable K/V Backend

    KES stores the secret keys at the Vault K/V backend. Vault provides two K/V engines, v1 and v2.

    MinIO recommends the K/V v1 engine.

    The following command enables the K/V v1 secret engine:

    $ vault secrets enable -version=1 kv

    The following command enables the K/V v2 secret engine:

    $ vault secrets enable -version=2 kv
    Note that the Vault policy for KES depends on the chosen K/V engine version. The v2 engine requires slightly different policy rules compared to the v1 engine. For more information about migrating from v1 to v2 see upgrading from v1.

  9. Create Vault Policy

    The Vault policy defines the API paths the KES server can access.

    • For v1

      The following kes-policy.hcl policy should be used for the K/V v1 backend:

      path "kv/*" {
   capabilities = [ "create", "read", "delete" ]
}

    • For v2 T The following kes-policy.hcl policy should be used for the K/V v2 backend:

      path "kv/data/*" {
   capabilities = [ "create", "read" ]
}
path "kv/metadata/*" {
   capabilities = [ "list", "delete" ]       
}

    The following command creates the policy at Vault:

    $ vault policy write kes-policy kes-policy.hcl

  10. Enable AppRole Authentication

    This step allows the KES server to authenticate to Vault. For this tutorial, we use the AppRole authentication method.

    $ vault auth enable approle

  11. Create KES Role

    The following command adds a new role kes-server at Vault:

    $ vault write auth/approle/role/kes-server token_num_uses=0  secret_id_num_uses=0  period=5m

  12. Bind Policy to Role

    The following command binds kes-server role to the key-policy:

    $ vault write auth/approle/role/kes-server policies=kes-policy

  13. Generate AppRole ID

    Request an AppRole ID for the KES server:

    $ vault read auth/approle/role/kes-server/role-id

  14. Generate AppRole Secret

    Request an AppRole secret for the KES server:

    $ vault write -f auth/approle/role/kes-server/secret-id

    The AppRole secret prints as secret_id. You can ignore the secret_id_accessor.

KES Server Setup

  1. Generate KES Server Private Key & Certificate

    The following command generates a new TLS private key server.key and a self-signed X.509 certificate server.cert that is issued for the IP 127.0.0.1 and DNS name localhost (as SAN). Customize the command to match your setup.

     kes tool identity new --server --key server.key --cert server.cert --ip "127.0.0.1" --dns localhost

    Any other tooling for X.509 certificate generation works as well. For example, you could use openssl:

    $ openssl ecparam -genkey -name prime256v1 | openssl ec -out server.key

$ openssl req -new -x509 -days 30 -key server.key -out server.cert \
    -subj "/C=/ST=/L=/O=/CN=localhost" -addext "subjectAltName = IP:127.0.0.1"

  2. Generate Client Credentials

    The following command generates a new TLS private/public key pair for the client application to use for the KES Server:

    $ kes identity new --key=client.key --cert=client.crt MyApp

  Private key:  client.key
  Certificate:  client.crt
  Identity:     02ef5321ca409dbc7b10e7e8ee44d1c3b91e4bf6e2198befdebee6312745267b

    The identity 02ef5321ca409dbc7b10e7e8ee44d1c3b91e4bf6e2198befdebee6312745267b is a unique fingerprint of the public key in client.crt. You can re-compute it anytime:

    $ kes identity of client.crt

  Identity:  02ef5321ca409dbc7b10e7e8ee44d1c3b91e4bf6e2198befdebee6312745267b

  3. Configure KES Server

    Create the KES server configuration file: config.yml.

    Make sure that the identity in the policy section matches the client.crt identity. Add the approle role_id and secret_id obtained earlier.

    address: 0.0.0.0:7373 # Listen on all network interfaces on port 7373

admin:
  identity: disabled  # We disable the admin identity since we don't need it in this guide 

tls:
  key: private.key    # The KES server TLS private key
  cert: public.crt    # The KES server TLS certificate

policy:
  my-app: 
    allow:
    - /v1/key/create/my-key*
    - /v1/key/generate/my-key*
    - /v1/key/decrypt/my-key*
    identities:
    - 02ef5321ca409dbc7b10e7e8ee44d1c3b91e4bf6e2198befdebee6312745267b # Use the identity of your client.crt

keystore:
   vault:
     endpoint: https://127.0.0.1:8200
     version:  v1 # The K/V engine version - either "v1" or "v2".
     approle:
       id:     "" # Your AppRole ID
       secret: "" # Your AppRole Secret
       retry:  15s
     status:
       ping: 10s
     tls:
       ca: vault.crt # Manually trust the vault certificate since we use self-signed certificates

  4. Start KES Server

    $ kes server --config config.yml --auth off
    Linux Swap Protection:

    In Linux environments, KES can use the mlock syscall to prevent the OS from writing in-memory data to disk (swapping). This prevents leaking sensitive data.

    Use the following command to allow KES to use the mlock syscall without running with root privileges:

    $ sudo setcap cap_ipc_lock=+ep $(readlink -f $(which kes))

    Start a KES server instance with memory protection:

    $ kes server --config config.yml --auth off --mlock

KES CLI Access

  1. Set KES_SERVER Endpoint

    The following environment variable specifies the server the KES CLI should talk to:

    $ export KES_SERVER=https://127.0.0.1:7373

  2. Define the Client Credentials

    The following environment variables set the access credentials the client uses to talk to a KES server:

    $ export KES_CLIENT_CERT=client.crt

    $ export KES_CLIENT_KEY=client.key

  3. Test the Configuration

    Perform any API operation allowed by the policy we assigned above.

    For example, create a key:

    $ kes key create my-key-1

    Use the key to generate a new data encryption key:

    $ kes key dek my-key-1
{
  plaintext : UGgcVBgyQYwxKzve7UJNV5x8aTiPJFoR+s828reNjh0=
  ciphertext: eyJhZWFkIjoiQUVTLTI1Ni1HQ00tSE1BQy1TSEEtMjU2IiwiaWQiOiIxMTc1ZjJjNDMyMjNjNjNmNjY1MDk5ZDExNmU3Yzc4NCIsIml2IjoiVHBtbHpWTDh5a2t4VVREV1RSTU5Tdz09Iiwibm9uY2UiOiJkeGl0R3A3bFB6S21rTE5HIiwiYnl0ZXMiOiJaaWdobEZrTUFuVVBWSG0wZDhSYUNBY3pnRWRsQzJqWFhCK1YxaWl2MXdnYjhBRytuTWx0Y3BGK0RtV1VoNkZaIn0=
}

    To run KES locally for testing purposes, use the -k or -insecure flag to generate a new data encryption key:

    $ kes key dek my-key-1 -k
{
  plaintext : UGgcVBgyQYwxKzve7UJNV5x8aTiPJFoR+s828reNjh0=
  ciphertext: eyJhZWFkIjoiQUVTLTI1Ni1HQ00tSE1BQy1TSEEtMjU2IiwiaWQiOiIxMTc1ZjJjNDMyMjNjNjNmNjY1MDk5ZDExNmU3Yzc4NCIsIml2IjoiVHBtbHpWTDh5a2t4VVREV1RSTU5Tdz09Iiwibm9uY2UiOiJkeGl0R3A3bFB6S21rTE5HIiwiYnl0ZXMiOiJaaWdobEZrTUFuVVBWSG0wZDhSYUNBY3pnRWRsQzJqWFhCK1YxaWl2MXdnYjhBRytuTWx0Y3BGK0RtV1VoNkZaIn0=
}

Advanced Configuration

These additional configuration steps may solve specific problems.

Multi-Tenancy with K/V prefixes

Vault can serve as backend for multiple, isolated KES tenants. Each KES tenant can consist of N replicas. There can be M KES tenants connected to the same Vault server/cluster.

This means N × M KES server instances can connect to a single Vault.

In these configurations, each KES tenant has a separate prefix at the K/V secret engine. For each KES tenant, there must be a corresponding Vault policy.

  • For K/V v1:

    path "kv/<tenant-name>/*" {
   capabilities = [ "create", "read", "delete" ]
}

  • For K/V v2:

    path "kv/data/<tenant-name>/*" {
  capabilities = [ "create", "read" ]
}
path "kv/metadata/<tenant-name>/*" {
  capabilities = [ "list", "delete" ]       
}

Create a different configuration file for each KES tenant. The file contains the Vault K/V prefix for the tenant to use.

keystore:
   vault:
     endpoint: https://127.0.0.1:8200
     prefix: <tenant-name>
     approle:
       id:     "" # Your AppRole ID
       secret: "" # Your AppRole Secret
       retry:  15s
     status:
       ping: 10s
     tls:
       ca: vault.crt # Manually trust the vault certificate since we use self-signed certificates

Multi-Tenancy with Vault Namespaces

Vault can serve as the backend for multiple, isolated KES tenants.

Vault can serve as backend for multiple, isolated KES tenants. Each KES tenant can consist of N replicas. There can be M KES tenants connected to the same Vault server/cluster.

This means N × M KES server instances can connect to a single Vault.

Therefore, each KES tenant has a separate prefix at the K/V secret engine. For each KES tenant there has to be a corresponding Vault policy.

  • For K/V v1:

    path "kv/<tenant-name>/*" {
   capabilities = [ "create", "read", "delete" ]
}

  • For K/V v2:

    path "kv/data/<tenant-name>/*" {
   capabilities = [ "create", "read" ]
}
path "kv/metadata/<tenant-name>/*" {
   capabilities = [ "list", "delete" ]       
}

Use a different configuration file for each KES tenant. The file contains the Vault namespace which the KES tenant should use.

keystore:
   vault:
     endpoint: https://127.0.0.1:8200
     namespace: <vault-namespace>
     approle:
       id:     "" # Your AppRole ID
       secret: "" # Your AppRole Secret
       retry:  15s
     status:
       ping: 10s
     tls:
       ca: vault.crt # Manually trust the vault certificate since we use self-signed certificates

References