Documentation

Deploy a MinIO Tenant

This procedure documents deploying a MinIO Tenant onto a stock Kubernetes cluster using the MinIO Operator Console.

The MinIO Operator Console is designed for deploying multi-node distributed MinIO Deployments.

Deploying Single-Node topologies requires additional configurations not covered in this documentation. You can alternatively use a simple Kubernetes YAML object to describe a Single-Node topology for local testing and evaluation as necessary.

MinIO does not recommend nor support single-node deployment topologies for production environments.

The Operator Console provides a rich user interface for deploying and managing MinIO Tenants on Kubernetes infrastructure. Installing the MinIO Kubernetes Operator automatically installs and configures the Operator Console.

This documentation assumes familiarity with all referenced Kubernetes concepts, utilities, and procedures. While this documentation may provide guidance for configuring or deploying Kubernetes-related resources on a best-effort basis, it is not a replacement for the official Kubernetes Documentation.

Prerequisites

MinIO Kubernetes Operator

The procedures on this page requires a valid installation of the MinIO Kubernetes Operator and assumes the local host has a matching installation of the MinIO Kubernetes Operator. This procedure assumes the latest stable Operator, version 5.0.15.

See Deploy the MinIO Operator for complete documentation on deploying the MinIO Operator.

AKS Cluster with Azure Virtual Machines

This procedure assumes an existing AKS cluster with at least four Azure virtual machines (VM). The Azure VMs should have matching machine types and configurations to ensure predictable performance with MinIO.

MinIO provides hardware guidelines for selecting the appropriate EC2 instance class and size. MinIO strongly recommends selecting VM instances with support for Premium SSDs and at least 25Gbps Network bandwidth as a baseline for performance.

For more complete information on Azure Virtual Machine types and Storage resources, see Sizes for virtual machines in Azure and Azure managed disk types

Persistent Volumes

Exclusive access to drives

MinIO requires exclusive access to the drives or volumes provided for object storage. No other processes, software, scripts, or persons should perform any actions directly on the drives or volumes provided to MinIO or the objects or files MinIO places on them.

Unless directed by MinIO Engineering, do not use scripts or tools to directly modify, delete, or move any of the data shards, parity shards, or metadata files on the provided drives, including from one drive or node to another. Such operations are very likely to result in widespread corruption and data loss beyond MinIO’s ability to heal.

MinIO can use any Kubernetes Persistent Volume (PV) that supports the ReadWriteOnce access mode. MinIO’s consistency guarantees require the exclusive storage access that ReadWriteOnce provides. Additionally, MinIO recommends setting a reclaim policy of Retain for the PVC StorageClass. Where possible, configure the Storage Class, CSI, or other provisioner underlying the PV to format volumes as XFS to ensure best performance.

For Kubernetes clusters where nodes have Direct Attached Storage, MinIO strongly recommends using the DirectPV CSI driver. DirectPV provides a distributed persistent volume manager that can discover, format, mount, schedule, and monitor drives across Kubernetes nodes. DirectPV addresses the limitations of manually provisioning and monitoring local persistent volumes.

MinIO Tenants on AKS should use the Azure Disks CSI driver to provision the necessary underlying persistent volumes. MinIO strongly recommends SSD-backed disk types for best performance. For more information on AKS disk types, see Azure disk types.

Deploy a Tenant using the MinIO Operator Console

To deploy a tenant from the MinIO Operator Console, complete the following steps in order:

1) Access the MinIO Operator Console

2) Complete the Tenant Setup

3) The Configure Section

4) The Images Section

5) The Pod Placement Section

6) The Identity Provider Section

7) The Security Section

8) The Encryption Section

9) Deploy and View the Tenant

10) Connect to the Tenant

1) Access the MinIO Operator Console

Port Forwarding

The Operator Console service does not automatically bind or expose itself for external access on the Kubernetes cluster. Instead, configure a network control plane component, such as a load balancer or ingress, to grant external access.

For testing purposes or short-term access, expose the Operator Console service through a NodePort using the following patch:

kubectl patch service -n minio-operator console -p '
{
    "spec": {
        "ports": [
            {
                "name": "http",
                "port": 9090,
                "protocol": "TCP",
                "targetPort": 9090,
                "nodePort": 30090
            },
            {
                "name": "https",
                "port": 9443,
                "protocol": "TCP",
                "targetPort": 9443,
                "nodePort": 30433
            }
        ],
        "type": "NodePort"
    }
}'

The patch command should output service/console patched. You can now access the service through ports 30433 (HTTPS) or 30090 (HTTP) on any of your Kubernetes worker nodes.

For example, a Kubernetes cluster with the following Operator nodes might be accessed at https://172.18.0.2:30443:

kubectl get nodes -o custom-columns=IP:.status.addresses[:]
IP
map[address:172.18.0.5 type:InternalIP],map[address:k3d-MINIO-agent-3 type:Hostname]
map[address:172.18.0.6 type:InternalIP],map[address:k3d-MINIO-agent-2 type:Hostname]
map[address:172.18.0.2 type:InternalIP],map[address:k3d-MINIO-server-0 type:Hostname]
map[address:172.18.0.4 type:InternalIP],map[address:k3d-MINIO-agent-1 type:Hostname]
map[address:172.18.0.3 type:InternalIP],map[address:k3d-MINIO-agent-0 type:Hostname]

Use the following command to retrieve the JWT token necessary for logging into the Operator Console:

kubectl get secret/console-sa-secret -n minio-operator -o json | jq -r '.data.token' | base64 -d

If your local host does not have the jq utility installed, you can run the kubectl part of this command (before | jq) and locate the data.token section of the output.

Open your browser to the appropriate URL and enter the JWT Token into the login page. You should see the Tenants page.

Select + Create Tenant to start creating a MinIO Tenant.

2) Complete the Tenant Setup

The Setup pane displays core configuration settings for the MinIO Tenant.

Settings marked with an asterisk * are required:

Field

Description

Name

The name of the MinIO Tenant

Namespace

The Kubernetes Namespace in which to deploy the tenant. You can create the namespace by selecting the plus + icon if it does not exist.

The Operator supports at most one MinIO Tenant per namespace.

Storage Class

Specify the Kubernetes Storage Class the Operator uses when generating Persistent Volume Claims for the Tenant.

Ensure the specified storage class has sufficient available Persistent Volume resources to match each generated Persistent Volume Claim.

Specify the AKS persistent disk type to use for this tenant. The AKS CSI Driver provides the following storage classes by default:

  • managed-csi (Standard SSD)

  • managed-csi-premium (Premium SSD)

You can create additional Storage Classes to represent other supported persistent disk types. See Create a custom storage class for more information.

Number of Servers

The total number of MinIO server pods to deploy in the Tenant. The Operator enforces a minimum of four server pods per tenant.

The Operator by default uses pod anti-affinity, such that the Kubernetes cluster must have at least one worker node per MinIO server pod. Use the Pod Placement pane to modify the pod scheduling settings for the Tenant.

Drives per Server

The number of storage volumes (Persistent Volume Claims) the Operator requests per Server.

The Operator displays the Total Volumes under the Resource Allocation section. The Operator generates an equal number of PVC plus two for supporting Tenant services (Metrics and Log Search).

The specified Storage Class must correspond to a set of Persistent Volumes sufficient in number to match each generated PVC.

For deployments using the AKS CSI Driver, the Operator provisions Persistent Volume Claims which result in the creation of Azure Disks of the specified Storage Class equal to Number of Drives per Server X Number of Servers.

Total Size

The total raw storage size for the Tenant. Specify both the total storage size and the Unit of that storage. All storage units are in SI values, e.g. \(Gi = GiB = 1024^3\) bytes.

The Operator displays the Drive Capacity under the:guilabel:Resource Allocation section. The Operator sets this value as the requested storage capacity in each generated PVC.

The specified Storage Class must correspond to a set of Persistent Volumes sufficient in capacity to match each generated PVC.

Erasure Code Parity

The Erasure Code Parity to set for the deployment.

The Operator displays the selected parity and its effect on the deployment under the Erasure Code Configuration section. Erasure Code parity defines the overall resiliency and availability of data on the cluster. Higher parity values increase tolerance to drive or node failure at the cost of total storage. See Erasure Coding for more complete documentation.

CPU Request

Specify the desired number of CPUs to allocate per MinIO server pod.

Memory Request [Gi]

Specify the desired amount of memory (RAM) to allocate per MinIO server pod. See Memory for guidance on setting this value. MinIO requires a minimum of 2GiB of memory per worker.

The Kubernetes cluster must have worker nodes with sufficient free RAM to match the pod request.

Specify Limit

Toggle to ON to specify maximum CPU and memory limits.

Select Create to create the Tenant using the current configuration. While all subsequent sections are optional, MinIO recommends reviewing them prior to deploying the Tenant.

3) The Configure Section

The Configure section displays optional configuration settings for the MinIO Tenant and its supporting services.

Field

Description

Expose MinIO Service

The MinIO Operator by default directs the MinIO Tenant services to request an externally accessible IP address from the Kubernetes cluster Load Balancer if one is available to access the tenant.

Your Kubernetes distributions may include a load balancer that can respond to these requests. Installation and configuration of load balancers is out of the scope of this documentation.

Expose Console Service

Select whether the Tenant should request an IP address from the Load Balancer to access the Tenant’s Console.

Your Kubernetes distributions may include a load balancer that can respond to these requests. Installation and configuration of load balancers is out of the scope of this documentation.

Set Custom Domains

Toggle on to customize the domains allowed to access the tenant’s console and other tenant services.

Security Context

The MinIO Operator sets the Kubernetes Security Context for pods to a default of 1000 for User, Group, and FsGroup. The FSGroupChangePolicy defaults to Always. MinIO does not run the pod using the root user.

You can modify the Security Context to direct MinIO to run using a different User, Group,FsGroup ID, and FSGroupChangePolicy. You can also direct MinIO to run as the Root user.

Custom Runtime Configurations

Toggle on to customize the Runtime Class for the tenant to use.

Additional Environment Variables

Enter any additional the key:value pairs to use as environment variables for the tenant.

4) The Images Section

The Images section displays container image settings used by the MinIO Tenant.

Field

Description

MinIO

The container image to use for the MinIO Server. See the MinIO Quay or the MinIO DockerHub repositories for a list of valid tags.

KES Image

The container image to use for MinIO KES.

Use a private container registry

If the tenant requires a private container registry, toggle to ON, then specify the location and credentials for the private registry.

5) The Pod Placement Section

The Pod Placement section displays pod scheduler settings for the MinIO Tenant.

Field

Description

None

Disables pod scheduling constraints for the tenant. This allows Kubernetes to schedule multiple Tenant pods onto the same node.

This may decrease resiliency, as a single Kubernetes worker can host multiple MinIO pods. If that worker is down or lost, objects may also be unavailable or lost.

Consider using this setting only in early development or sandbox environments with a limited number of worker nodes.

Default (Pod Anti-Affinity)

Directs the Operator to set anti-affinity settings such that no Kubernetes worker can host more than one MinIO server pod for this Tenant.

Node Selector

Directs the operator to set a Node Selector such that pods only deploy onto Kubernetes workers whose labels match the selector.

Tolerations

Specify any required tolerations for this tenant’s pods.

6) The Identity Provider Section

The Identity Provider section displays the Identity Provider settings for the MinIO Tenant. This includes configuring an external IDP such as OpenID or Active Directory / LDAP.

Field

Description

Built-In

Configure additional internal MinIO users for the Operator to create as part of deploying the Tenant.

OpenID

Configure an OpenID Connect-compatible service as an external Identity Provider (e.g. Keycloak, Okta, Google, Facebook, Dex) to manage MinIO users.

Active Directory

Configure an Active Directory or OpenLDAP service as the external Identity Provider to manage MinIO users.

7) The Security Section

The Security section displays TLS certificate settings for the MinIO Tenant.

Field

Description

TLS

Enable or disable TLS for the MinIO Tenant.

AutoCert

Directs the Operator to generate Certificate Signing Requests for submission to the Kubernetes TLS API.

The MinIO Tenant uses the generated certificates for enabling and establishing TLS connections.

Custom Certificates

When enabled, you can upload custom TLS certificates for MinIO to use for server and client credentials.

MinIO supports Server Name Indication (SNI) such that the Tenant can select the appropriate TLS certificate based on the request hostname and the certificate Subject Alternative Name.

MinIO also supports uploading Certificate Authority certificates for validating client certificates minted by that CA.

Supported Secret Types

MinIO supports three types of secrets in Kubernetes.

  1. opaque

    Using private.key and public.crt files.

  2. tls

    Using tls.key and tls.crt files.

  3. cert-manager 1.7.x or later

    Running on Kubernetes 1.21 or later.

New in version Console: 0.23.1

A message displays under the certificate with the date of expiration and length of time until expiration.

The message adjusts depending on the length of time to expiration:

  • More than 30 days, the message text displays in gray.

  • Within 30 days, the message text changes to orange.

  • Within 10 days, the message text changes to red.

  • Within 24 hours, the message displays as an hour and minute countdown in red text.

  • After expiration, the message displays as EXPIRED.

8) The Encryption Section

The Encryption section displays the Server-Side Encryption (SSE) settings for the MinIO Tenant.

Enabling SSE also creates MinIO Key Encryption Service pods in the Tenant to facilitate SSE operations.

Field

Description

Vault

Configure HashiCorp Vault as the external KMS for storing root encryption keys. See Server-Side Object Encryption with KES for guidance on the displayed fields.

AWS

Configure AWS Secrets Manager as the external KMS for storing root encryption keys. See Server-Side Object Encryption with KES for guidance on the displayed fields.

Gemalto

Configure Gemalto (Thales Digital Identity and Security) as the external KMS for storing root encryption keys. See Thales CipherTrust Manager (formerly Gemalto KeySecure) for guidance on the displayed fields.

GCP

Configure Google Cloud Platform Secret Manager as the external KMS for storing root encryption keys. See Server-Side Object Encryption with KES for guidance on the displayed fields.

Azure

Configure Azure Key Vault as the external KMS for storing root encryption keys. See Server-Side Object Encryption with KES for guidance on the displayed fields.

9) Deploy and View the Tenant

Select Create at any time to begin the deployment process. The MinIO Operator displays the root user credentials once as part of deploying the Tenant. Copy these credentials to a secure location.

You can monitor the Tenant creation process from the Tenants view. The State column updates throughout the deployment process.

Tenant deployment can take several minutes to complete. Once the State reads as Initialized, select the Tenant to view its details.

Each tab provides additional details or configuration options for the MinIO Tenant.

  • METRICS - Displays metrics collected from the MinIO Tenant.

  • SECURITY - Provides TLS-related configuration options.

  • POOLS - Supports expanding the tenant by adding more Server Pools.

  • LICENSE - Enter your SUBNET license.

10) Connect to the Tenant

The MinIO Operator creates services for the MinIO Tenant.

Use the kubectl get svc -n NAMESPACE command to review the deployed services:

kubectl get svc -n minio-tenant-1
NAME                               TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
minio                              LoadBalancer   10.97.114.60     <pending>     443:30979/TCP    2d3h
minio-tenant-1-console             LoadBalancer   10.106.103.247   <pending>     9443:32095/TCP   2d3h
minio-tenant-1-hl                  ClusterIP      None             <none>        9000/TCP         2d3h
minio-tenant-1-log-hl-svc          ClusterIP      None             <none>        5432/TCP         2d3h
minio-tenant-1-log-search-api      ClusterIP      10.103.5.235     <none>        8080/TCP         2d3h
minio-tenant-1-prometheus-hl-svc   ClusterIP      None             <none>        9090/TCP         7h39m
  • The minio service corresponds to the MinIO Tenant service. Applications should use this service for performing operations against the MinIO Tenant.

  • The *-console service corresponds to the MinIO Console. Administrators should use this service for accessing the MinIO Console and performing administrative operations on the MinIO Tenant.

The remaining services support Tenant operations and are not intended for consumption by users or administrators.

By default each service is visible only within the Kubernetes cluster. Applications deployed inside the cluster can access the services using the CLUSTER-IP.

Applications external to the Kubernetes cluster can access the services using the EXTERNAL-IP. This value is only populated for Kubernetes clusters configured for Ingress or a similar network access service. Kubernetes provides multiple options for configuring external access to services.

See the Kubernetes documentation on Publishing Services (ServiceTypes) and Ingress for more complete information on configuring external access to services.