Deploy the MinIO Operator

Overview

MinIO is a Kubernetes-native high performance object store with an S3-compatible API. The MinIO Kubernetes Operator supports deploying MinIO Tenants onto private and public cloud infrastructures (“Hybrid” Cloud).

The following procedure installs the latest stable version (5.0.15) of the MinIO Operator on Kubernetes infrastructure.

The MinIO Operator installs a Custom Resource Definition (CRD) to support describing MinIO tenants as a Kubernetes object. See the MinIO Operator CRD Reference for complete documentation on the MinIO CRD.

This documentation assumes familiarity with referenced Kubernetes concepts, utilities, and procedures. While this documentation may provide guidance for configuring or deploying Kubernetes-related resources on a best-effort basis, it is not a replacement for the official Kubernetes Documentation.

MinIO Operator Components

The MinIO Operator exists in its own namespace. Within the Operator’s namespace, the MinIO Operator utilizes two pods:

The Operator pod for the base Operator functions to deploy, manage, modify, and maintain tenants.
Console pod for the Operator’s Graphical User Interface, the Operator Console.

When you use the Operator to create a tenant, the tenant must have its own namespace. Within that namespace, the Operator generates the pods required by the tenant configuration.

Each pod runs three containers:

MinIO Container that runs all of the standard MinIO functions, equivalent to basic MinIO installation on baremetal. This container stores and retrieves objects in the provided mount points (persistent volumes).
InitContainer that only exists during the launch of the pod to manage configuration secrets during startup. Once startup completes, this container terminates.
SideCar container that monitors configuration secrets for the tenant and updates them as they change. This container also monitors for root credentials and creates an error if it does not find root credentials.

Starting with v5.0.6, the MinIO Operator supports custom init containers for additional pod initialization that may be required for your environment.

The tenant utilizes Persistent Volume Claims to talk to the Persistent Volumes that store the objects.

A diagram of the namespaces and pods used by or maintained by the MinIO Operator.

Prerequisites

Kubernetes Version 1.21.0

Important

MinIO strongly recommends upgrading Production clusters running End-Of-Life Kubernetes APIs.

Starting with v5.0.0, MinIO requires Kubernetes 1.21.0 or later for both the infrastructure and the kubectl CLI tool.

New in version Operator: 5.0.6

For Kubernetes 1.25.0 and later, MinIO supports deploying in environments with the Pod Security admission (PSA) restricted policy enabled.

Kustomize and `kubectl`

Kustomize is a YAML-based templating tool that allows you to define Kubernetes resources in a declarative and repeatable fashion. Kustomize is included with the kubectl command line tool.

This procedure assumes that your local host machine has both the matching version of kubectl for your Kubernetes cluster and the necessary access to that cluster to create new resources.

The default MinIO Operator Kustomize template provides a starting point for customizing configurations for your local environment. You can modify the default Kustomization file or apply your own patches to customize the Operator deployment for your Kubernetes cluster.

Kubernetes TLS Certificate API

Changed in version Operator: v.5.0.0

The MinIO Operator manages TLS Certificate Signing Requests (CSR) using the Kubernetes certificates.k8s.io TLS certificate management API to create signed TLS certificates in the following circumstances:

When autoCert is enabled.
For the MinIO Console when the MINIO_CONSOLE_TLS_ENABLE environment variable is set to on.
For STS service when OPERATOR_STS_ENABLED environment variable is set to on.
For retrieving the health of the cluster.

The MinIO Operator reads certificates inside the operator-ca-tls secret and syncs this secret within the tenant namespace to trust private certificate authorities, such as when using cert-manager.

For any of these circumstances, the MinIO Operator requires that the Kubernetes kube-controller-manager configuration include the following configuration settings:

--cluster-signing-key-file - Specify the PEM-encoded RSA or ECDSA private key used to sign cluster-scoped certificates.
--cluster-signing-cert-file - Specify the PEM-encoded x.509 Certificate Authority certificate used to issue cluster-scoped certificates.

The Kubernetes TLS API uses the CA signature algorithm for generating new TLS certificate. MinIO recommends ECDSA (e.g. NIST P-256 curve) or EdDSA (e.g. Curve25519) TLS private keys/certificates due to their lower computation requirements compared to RSA. See Supported TLS Cipher Suites for a complete list of supported TLS Cipher Suites.

If the Kubernetes cluster is not configured to respond to a generated CSR, the Operator cannot complete initialization. Some Kubernetes providers do not specify these configuration values by default.

To check whether the kube-controller-manager specifies the cluster signing key and certificate files, use the following command:

kubectl get pod kube-controller-manager-$CLUSTERNAME-control-plane \
  -n kube-system -o yaml

Replace $CLUSTERNAME with the name of the Kubernetes cluster.

Confirm that the output contains the highlighted lines. The output of the example command above may differ from the output in your terminal:

 spec:
 containers:
 - command:
     - kube-controller-manager
     - --allocate-node-cidrs=true
     - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
     - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
     - --bind-address=127.0.0.1
     - --client-ca-file=/etc/kubernetes/pki/ca.crt
     - --cluster-cidr=10.244.0.0/16
     - --cluster-name=my-cluster-name
     - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
     - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
 ...

Important

The MinIO Operator automatically generates TLS certificates for all MinIO Tenant pods using the specified Certificate Authority (CA). Clients external to the Kubernetes cluster must trust the Kubernetes cluster CA to connect to the MinIO Operator or MinIO Tenants.

Clients which cannot trust the Kubernetes cluster CA can disable TLS validation for connections to the MinIO Operator or a MinIO Tenant.

Alternatively, you can generate x.509 TLS certificates signed by a known and trusted CA and pass those certificates to MinIO Tenants. See Network Encryption (TLS) for more complete documentation.

Procedure

The following steps deploy Operator using Kustomize and a kustomization.yaml file from the MinIO Operator GitHub repository. To install Operator using a Helm chart, see Deploy Operator with Helm.

Install the MinIO Operator using Kustomize

The following procedure uses kubectl -k to install the Operator from the MinIO Operator GitHub repository. kubectl -k and kubectl --kustomize are aliases that perform the same command.

Important

If you use Kustomize to install the Operator, you must use Kustomize to manage or upgrade that installation. Do not use kubectl krew, a Helm chart, or similar methods to manage or upgrade a MinIO Operator installation deployed with Kustomize.

You can, however, use Kustomize to upgrade a previous version of Operator (5.0.14 or earlier) installed with the MinIO Kubernetes Plugin.

Install the latest version of Operator

kubectl apply -k "github.com/minio/operator?ref=v5.0.15"

The output resembles the following:

namespace/minio-operator created
customresourcedefinition.apiextensions.k8s.io/miniojobs.job.min.io created
customresourcedefinition.apiextensions.k8s.io/policybindings.sts.min.io created
customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io created
serviceaccount/console-sa created
serviceaccount/minio-operator created
clusterrole.rbac.authorization.k8s.io/console-sa-role created
clusterrole.rbac.authorization.k8s.io/minio-operator-role created
clusterrolebinding.rbac.authorization.k8s.io/console-sa-binding created
clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding created
configmap/console-env created
secret/console-sa-secret created
service/console created
service/operator created
service/sts created
deployment.apps/console created
deployment.apps/minio-operator created

Verify the Operator pods are running:

kubectl get pods -n minio-operator

The output resembles the following:

NAME                              READY   STATUS              RESTARTS   AGE
console-56c7d8bd89-485qh          1/1     Running   0          2m42s
minio-operator-6c758b8c45-nkhlx   1/1     Running   0          2m42s
minio-operator-6c758b8c45-dgd8n   1/1     Running   0          2m42s

In this example, the minio-operator pod is MinIO Operator and the console pod is the Operator Console.

You can modify your Operator deployment by applying kubectl patches. You can find examples for common configurations in the Operator GitHub repository.

(Optional) Configure access to the Operator Console service

The Operator Console service does not automatically bind or expose itself for external access on the Kubernetes cluster. You must instead configure a network control plane component, such as a load balancer or ingress, to grant that external access.

For testing purposes or short-term access, expose the Operator Console service through a NodePort using the following patch:

kubectl patch service -n minio-operator console -p '
{
    "spec": {
        "ports": [
            {
                "name": "http",
                "port": 9090,
                "protocol": "TCP",
                "targetPort": 9090,
                "nodePort": 30090
            },
            {
                "name": "https",
                "port": 9443,
                "protocol": "TCP",
                "targetPort": 9443,
                "nodePort": 30433
            }
        ],
        "type": "NodePort"
    }
}'

The patch command should output service/console patched. You can now access the service through ports 30433 (HTTPS) or 30090 (HTTP) on any of your Kubernetes worker nodes.

Verify the Operator installation

Check the contents of the specified namespace (minio-operator) to ensure all pods and services have started successfully.

kubectl get all -n minio-operator

The response should resemble the following:

NAME                                  READY   STATUS    RESTARTS   AGE
pod/console-56c7d8bd89-485qh          1/1     Running   0          5m20s
pod/minio-operator-6c758b8c45-nkhlx   1/1     Running   0          5m20s
pod/minio-operator-6c758b8c45-dgd8n   1/1     Running   0          5m20s

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/operator   ClusterIP   10.43.135.241   <none>        4221/TCP                        5m20s
service/sts        ClusterIP   10.43.117.251   <none>        4223/TCP                        5m20s
service/console    NodePort    10.43.235.38    <none>        9090:30090/TCP,9443:30433/TCP   5m20s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/console          1/1     1            1           5m20s
deployment.apps/minio-operator   2/2     2            2           5m20s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/console-56c7d8bd89          1         1         1       5m20s
replicaset.apps/minio-operator-6c758b8c45   2         2         2       5m20s

Retrieve the Operator Console JWT for login

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: console-sa-secret
  namespace: minio-operator
  annotations:
    kubernetes.io/service-account.name: console-sa
type: kubernetes.io/service-account-token
EOF
SA_TOKEN=$(kubectl -n minio-operator  get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode)
echo $SA_TOKEN

The output of this command is the JSON Web Token (JWT) login credential for Operator Console.

Log into the MinIO Operator Console
NodePort
If you configured the service for access through a NodePort, specify the hostname of any worker node in the cluster with that port as HOSTNAME:NODEPORT to access the Console.

For example, a deployment configured with a NodePort of 30090 and the following InternalIP addresses can be accessed at http://172.18.0.5:30090.
kubectl get nodes -o custom-columns=IP:.status.addresses[:] IP map[address:172.18.0.5 type:InternalIP],map[address:k3d-MINIO-agent-3 type:Hostname] map[address:172.18.0.6 type:InternalIP],map[address:k3d-MINIO-agent-2 type:Hostname] map[address:172.18.0.2 type:InternalIP],map[address:k3d-MINIO-server-0 type:Hostname] map[address:172.18.0.4 type:InternalIP],map[address:k3d-MINIO-agent-1 type:Hostname] map[address:172.18.0.3 type:InternalIP],map[address:k3d-MINIO-agent-0 type:Hostname]
Ingress or Load Balancer

If you configured the svc/console service for access through ingress or a cluster load balancer, you can access the Console using the configured hostname and port.

Port Forwarding
You can use kubectl port forward to temporary forward ports for the Console:
kubectl port-forward svc/console -n minio-operator 9090:9090
You can then use http://localhost:9090 to access the MinIO Operator Console.

Once you access the Console, use the Console JWT to log in. You can now deploy and manage MinIO Tenants using the Operator Console.

Deploy the MinIO Operator

Overview

MinIO Operator Components

Prerequisites

Kubernetes Version 1.21.0

Kustomize and kubectl

Kubernetes TLS Certificate API

Procedure

Install the MinIO Operator using Kustomize

Kustomize and `kubectl`