Documentation

Deploy the MinIO Operator

Overview

MinIO is a Kubernetes-native high performance object store with an S3-compatible API. The MinIO Kubernetes Operator supports deploying MinIO Tenants onto private and public cloud infrastructures (“Hybrid” Cloud).

The following procedure installs the latest stable version (5.0.11) of the MinIO Operator and MinIO Plugin on Kubernetes infrastructure:

  • The MinIO Operator installs a Custom Resource Definition (CRD) to support describing MinIO tenants as a Kubernetes object. See the MinIO Operator CRD Reference for complete documentation on the MinIO CRD.

  • The MinIO Kubernetes Plugin brings native support for deploying and managing MinIO tenants on a Kubernetes cluster using the kubectl minio command.

This documentation assumes familiarity with referenced Kubernetes concepts, utilities, and procedures. While this documentation may provide guidance for configuring or deploying Kubernetes-related resources on a best-effort basis, it is not a replacement for the official Kubernetes Documentation.

MinIO Operator Components

The MinIO Operator exists in its own namespace.

Within the Operator’s namespace, the MinIO Operator utilizes two pods: - The Operator pod for the base Operator functions to deploy, manage, modify, and maintain tenants. - Console pod for the Operator’s Graphical User Interface, the Operator Console.

When you use the Operator to create a tenant, the tenant must have its own namespace. Within that namespace, the Operator generates the pods required by the tenant configuration.

Each pod runs three containers:

  • MinIO Container that runs all of the standard MinIO functions, equivalent to basic MinIO installation on baremetal. This container stores and retrieves objects in the provided mount points (persistent volumes).

  • InitContainer that only exists during the launch of the pod to manage configuration secrets during startup. Once startup completes, this container terminates.

  • SideCar container that monitors configuration secrets for the tenant and updates them as they change. This container also monitors for root credentials and creates an error if it does not find root credentials.

Starting with v5.0.6, the MinIO Operator supports custom init containers for additional pod initialization that may be required for your environment.

The tenant utilizes Persistent Volume Claims to talk to the Persistent Volumes that store the objects.

A diagram of the namespaces and pods used by or maintained by the MinIO Operator.

Prerequisites

Kubernetes Version 1.19.0

Important

MinIO strongly recommends upgrading Production clusters running End-Of-Life Kubernetes APIs.

Starting with v4.0.0, the MinIO Operator and MinIO Kubernetes Plugin require Kubernetes 1.19.0 and later. The Kubernetes infrastructure and the kubectl CLI tool must have the same version of 1.19.0+.

Prior to v4.0.0, the MinIO Operator and Plugin required Kubernetes 1.17.0. You must upgrade your Kubernetes infrastructure to 1.19.0 or later to use the MinIO Operator or Plugin v4.0.0 or later.

Starting with v5.0.0, MinIO recommends Kubernetes 1.21.0 or later for both the infrastructure and the kubectl CLI tool.

New in version Operator: 5.0.6

For Kubernetes 1.25.0 and later, MinIO supports deploying in environments with the Pod Security admission (PSA) restricted policy enabled.

kubectl Configuration

This procedure assumes that your local host machine has both the correct version of kubectl for your Kubernetes cluster and the necessary access to that cluster to create new resources.

Kubernetes TLS Certificate API

Changed in version Operator: v.5.0.0

The MinIO Operator manages TLS Certificate Signing Requests (CSR) using the Kubernetes certificates.k8s.io TLS certificate management API to create signed TLS certificates in the following circumstances:

The MinIO Operator reads certificates inside the operator-ca-tls secret and syncs this secret within the tenant namespace to trust private certificate authorities, such as when using cert-manager.

For any of these circumstances, the MinIO Operator requires that the Kubernetes kube-controller-manager configuration include the following configuration settings:

  • --cluster-signing-key-file - Specify the PEM-encoded RSA or ECDSA private key used to sign cluster-scoped certificates.

  • --cluster-signing-cert-file - Specify the PEM-encoded x.509 Certificate Authority certificate used to issue cluster-scoped certificates.

The Kubernetes TLS API uses the CA signature algorithm for generating new TLS certificate. MinIO recommends ECDSA (e.g. NIST P-256 curve) or EdDSA (e.g. Curve25519) TLS private keys/certificates due to their lower computation requirements compared to RSA. See Supported TLS Cipher Suites for a complete list of supported TLS Cipher Suites.

If the Kubernetes cluster is not configured to respond to a generated CSR, the Operator cannot complete initialization. Some Kubernetes providers do not specify these configuration values by default.

To check whether the kube-controller-manager specifies the cluster signing key and certificate files, use the following command:

kubectl get pod kube-controller-manager-$CLUSTERNAME-control-plane \
  -n kube-system -o yaml
  • Replace $CLUSTERNAME with the name of the Kubernetes cluster.

Confirm that the output contains the highlighted lines. The output of the example command above may differ from the output in your terminal:

 spec:
 containers:
 - command:
     - kube-controller-manager
     - --allocate-node-cidrs=true
     - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
     - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
     - --bind-address=127.0.0.1
     - --client-ca-file=/etc/kubernetes/pki/ca.crt
     - --cluster-cidr=10.244.0.0/16
     - --cluster-name=my-cluster-name
     - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
     - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
 ...

Important

The MinIO Operator automatically generates TLS certificates for all MinIO Tenant pods using the specified Certificate Authority (CA). Clients external to the Kubernetes cluster must trust the Kubernetes cluster CA to connect to the MinIO Operator or MinIO Tenants.

Clients which cannot trust the Kubernetes cluster CA can disable TLS validation for connections to the MinIO Operator or a MinIO Tenant.

Alternatively, you can generate x.509 TLS certificates signed by a known and trusted CA and pass those certificates to MinIO Tenants. See Network Encryption (TLS) for more complete documentation.

Procedure

The following steps deploy Operator using the MinIO Kubernetes Plugin. To install Operator using a Helm chart, see Deploy Operator with Helm.

1) Install the MinIO Kubernetes Plugin

The MinIO Kubernetes Plugin provides a command for initializing the MinIO Operator.

You can install the MinIO plugin using either the Kubernetes Krew plugin manager or manually by downloading and installing the plugin binary to your local host:

Krew is a kubectl plugin manager developed by the Kubernetes SIG CLI group. See the krew installation documentation for specific instructions. You can use the Krew plugin for Linux, MacOS, and Windows operating systems.

You can use Krew to install the MinIO kubectl plugin using the following commands:

kubectl krew update
kubectl krew install minio

If you want to update the MinIO plugin with Krew, use the following command:

kubectl krew upgrade minio

You can validate the installation of the MinIO plugin using the following command:

kubectl minio version

The output should match 5.0.11.

You can download the MinIO kubectl plugin to your local system path. The kubectl CLI automatically discovers and runs compatible plugins.

The following code downloads the latest stable version 5.0.11 of the MinIO Kubernetes plugin and installs it to the system path:

curl https://github.com/minio/operator/releases/download/v5.0.11/kubectl-minio_5.0.11_linux_amd64 -o kubectl-minio
chmod +x kubectl-minio
mv kubectl-minio /usr/local/bin/

The mv command above may require sudo escalation depending on the permissions of the authenticated user.

Run the following command to verify installation of the plugin:

kubectl minio version

The output should display the Operator version as 5.0.11.

You can download the MinIO kubectl plugin to your local system path. The kubectl CLI automatically discovers and runs compatible plugins.

The following PowerShell command downloads the latest stable version 5.0.11 of the MinIO Kubernetes plugin and installs it to the system path:

Invoke-WebRequest -Uri "https://github.com/minio/operator/releases/download/v5.0.11/kubectl-minio_5.0.11_windows_amd64.exe" -OutFile "C:\kubectl-plugins\kubectl-minio.exe"

Ensure the path to the plugin folder is included in the Windows PATH.

Run the following command to verify installation of the plugin:

kubectl minio version

The output should display the Operator version as 5.0.11.

2) Initialize the MinIO Kubernetes Operator

Run the kubectl minio init command to initialize the MinIO Operator:

kubectl minio init

The command initializes the MinIO Operator with the following default settings:

Important

Document all arguments used when initializing the MinIO Operator.

3) Validate the Operator Installation

To verify the installation, run the following command:

kubectl get all --namespace minio-operator

If you initialized the Operator with a custom namespace, replace minio-operator with that namespace.

The output resembles the following:

NAME                                  READY   STATUS    RESTARTS   AGE
pod/console-59b769c486-cv7zv          1/1     Running   0          81m
pod/minio-operator-7976b4df5b-rsskl   1/1     Running   0          81m

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/console    ClusterIP   10.105.218.94    <none>        9090/TCP,9443/TCP   81m
service/operator   ClusterIP   10.110.113.146   <none>        4222/TCP,4233/TCP   81m

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/console          1/1     1            1           81m
deployment.apps/minio-operator   1/1     1            1           81m

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/console-59b769c486          1         1         1       81m
replicaset.apps/minio-operator-7976b4df5b   1         1         1       81m

4) Open the Operator Console

Port Forwarding

Note

Some Kubernetes deployments may experience issues with timeouts during port-forwarding operations with the Operator Console. Select the NodePorts section to view instructions for alternative access. You can alternatively configure your preferred Ingress to grant access to the Operator Console service. See https://github.com/kubernetes/kubectl/issues/1368 for more information.

Run the kubectl minio proxy command to temporarily forward traffic from the MinIO Operator Console service to your local machine:

kubectl minio proxy

The command output includes a required token for logging into the Operator Console.

MinIO Operator Console

You can deploy a new MinIO Tenant from the Operator Dashboard.

NodePorts

Use the following command to identify the NodePorts configured for the Operator Console. If your local host does not have the jq utility installed, you can run the first command and locate the spec.ports section of the output.

kubectl get svc/console -n minio-operator -o json | jq -r '.spec.ports'

The output resembles the following:

[
   {
      "name": "http",
      "nodePort": 31055,
      "port": 9090,
      "protocol": "TCP",
      "targetPort": 9090
   },
   {
      "name": "https",
      "nodePort": 31388,
      "port": 9443,
      "protocol": "TCP",
      "targetPort": 9443
   }
]

Use the http or https port depending on whether you deployed the Operator with Console TLS enabled via kubectl minio init --console-tls.

Append the nodePort value to the externally-accessible IP address of a worker node in your Kubernetes cluster.

Use the following command to retrieve the JWT token necessary for logging into the Operator Console:

kubectl get secret/console-sa-secret -n minio-operator -o json | jq -r '.data.token' | base64 -d