Google Cloud Secret Manager

This tutorial shows how to setup a KES server that uses GCP Secret Manager as a persistent key store.

K E S C l i e n t K E S S e r v e r G C P S e c r e t M a n a g e r

Google Cloud Secret Manager

The Google Cloud Secret Manager is a key-value store for secrets, such as passwords, access tokens, and cryptographic keys.

  1. Login in to the GCP console

  2. Create a new project or select an existing project

  3. Enable the Secret Manager service if not already enabled for your project

  4. Go to GCP IAM for service accounts and create a new service account for KES. KES uses this service account to authenticate to GCP and access the Secret Manager.

  5. Assign one or more roles to the new account

    If you want to get started quickly, assign the Secret Manager Admin role. However, this grants more permissions than KES needs.

    Alternatively, create a new role for KES with the minimal permissions required:

  6. Create a key for the service account via Actions - Create Key.

    Use the JSON key format.

    GCP let’s you download a JSON file with the following structure:

      "type":           "service_account",
      "project_id":     "<your-project-id>",
      "private_key_id": "<your-private-key-id>",
      "private_key":    "-----BEGIN PRIVATE KEY-----\n ... -----END PRIVATE KEY-----\n",
      "client_email":   "<your-service-account>@<your-project-id>",
      "client_id":      "<your-client-id>"

    Use this credentials file to configure KES to authenticate to GCP and access the SecretManager.

KES Server setup

The KES Server requires a TLS private key and certificate.

The KES server is secure-by-default and can only run with TLS. This tutorial uses self-signed certificates for simplicity.

For a production setup we highly recommend to use a certificate signed by trusted Certificate Authority. This can be either your internal CA or a public CA such as Let’s Encrypt.
  1. Generate a TLS private key and certificate for the KES server

    The following command generates a new TLS private key server.key and a self-signed X.509 certificate server.cert that is issued for the IP and DNS name localhost (as SAN). Customize the command to match your setup.

    kes tool identity new --server --key server.key --cert server.cert --ip "" --dns localhost

    Any other tooling for X.509 certificate generation works as well. For example, you could use openssl:

    $ openssl ecparam -genkey -name prime256v1 | openssl ec -out server.key
    $ openssl req -new -x509 -days 30 -key server.key -out server.cert \
        -subj "/C=/ST=/L=/O=/CN=localhost" -addext "subjectAltName = IP:"
  2. Create a private key and certificate for the application

    kes tool identity new --key=app.key --cert=app.cert app

    You can compute the app identity at any time.

    kes tool identity of app.cert
  3. Create the config file server-config.yml

    root:    disabled  # We disable the root identity since we don't need it in this guide 
      key : server.key
      cert: server.cert
        - /v1/key/create/my-app*
        - /v1/key/generate/my-app*
        - /v1/key/decrypt/my-app*
        - ${APP_IDENTITY}
          project_id: "<your-project-id>"                  # Use your GCP project ID
            client_email: "<your-client-email>"            # Use the client email from your GCP credentials file
            client_id: "<your-client-id>"                  # Use the client ID from your GCP credentials file
            private_key_id: "<your-private-key-id"         # Use the private key ID from your GCP credentials file
            private_key: "-----BEGIN PRIVATE KEY----- ..." # Use the private key from your GCP credentials file
  4. Start a KES server in a new window/tab:


    export APP_IDENTITY=$(kes tool identity of app.cert)
    kes server --config=server-config.yml --auth=off
    The command uses --auth=off because our root.cert and app.cert certificates are self-signed.


    The following instructions use Podman to manage the containers. You can also use Docker.

    Modify addresses and file paths as needed for your deployment.

    sudo podman pod create  \
      -p 9000:9000 -p 9001:9001 -p 7373:7373  \
      -v ~/minio-kes-gcp/certs:/certs  \
      -v ~/minio-kes-gcp/minio:/mnt/minio  \
      -v ~/minio-kes-gcp/config:/etc/default/  \
      -n minio-kes-gcp
    sudo podman run -dt  \
      --cap-add IPC_LOCK  \
      --name kes-server  \
      --pod "minio-kes-gcp"  \
      -e KES_SERVER=  \
      -e KES_CLIENT_KEY=/certs/kes-server.key  \
      -e KES_CLIENT_CERT=/certs/kes-server.cert  \ server  \
        --auth  \
        --config=/etc/default/kes-config.yaml  \
    sudo podman run -dt  \
      --name minio-server  \
      --pod "minio-kes-gcp"  \
      -e "MINIO_CONFIG_ENV_FILE=/etc/default/minio"  \ server  \
        --console-address ":9001"

    You can verify the status of the containers using the following command. The command should show three pods, one for the Pod, one for KES, and one for MinIO.

    sudo podman container list
  5. In the other window or tab, connect to the server

    export KES_CLIENT_CERT=app.cert
    export KES_CLIENT_KEY=app.key
    kes key create -k my-app-key
    export APP_IDENTITY=$(kes tool identity of app.cert)
    kes server --config=server-config.yml --auth=off
    The command uses --auth=off because our root.cert and app.cert certificates are self-signed.

    Now, if you go to the GCP Secret Manager, you should see a secret key named my-app-key.

  6. Derive and decrypt data keys from the previously created my-app-key

    kes key derive -k my-app-key
      plaintext : ...
      ciphertext: ...
    kes key decrypt -k my-app-key <base64-ciphertext>

Using KES with a MinIO Server

MinIO Server requires KES to enable server-side data encryption.

See the KES for MinIO instruction guide for additional steps needed to use your new KES Server with a MinIO Server.

Configuration References

The following section describes the Key Encryption Service (KES) configuration settings to use Google Cloud Secret Manager as the root KMS to store external keys, such as the keys used for Server-Side Encryption on a MinIO Server.

MinIO Server Requires Expanded Permissions:
Starting with MinIO Server RELEASE.2023-02-17T17-52-43Z, MinIO requires expanded KES permissions for functionality. The example configuration in this section contains all required permissions.