MinIO for Kubernetes

Customers run MinIO on Kubernetes for three reasons:

01.

Deploy, manage and secure S3-like infrastructure where MinIO provides object storage and Kubernetes provides compute infrastructure.

02.

Turnkey multi-cluster deployment and management of DevOps tools, providing freedom to innovate without lock-in or disruption while ensuring a consistent developer experience across locations, clouds and platforms.

03.

Using MinIO for Kubernetes storage provides control over the software stack with flexibility to avoid cloud lock-in and provide consistent object storage across hybrid and multi-cloud.

MinIO is high-performance Kubernetes-native object storage that is compatible with the S3 API. We recommend using MinIO wherever you need complete S3 API functionality for object storage on Kubernetes. MinIO provides a single global namespace and a consistent object storage interface across multiple cloud providers, on premise and at the edge.

MinIO natively integrates with Kubernetes to streamline operations for large scale multi-tenant object storage as a service, across multiple clouds and at the edge. MinIO can be managed through multiple tools. In Kubernetes environments, MinIO Operator and kubectl plugin simplify deployment and management for DevOps and infrastructure teams.

With Kubernetes as its engine, MinIO is able to run anywhere Kubernetes does - which in the modern, cloud-native world, is essentially everywhere. See the following Kubernetes powered environments with detailed information on the integration:

VMware Tanzu OpenShift SUSE Rancher EKS AKS GKE

While MinIO is integrated with other Kubernetes environments, we have always supported the developer who is interested in creating customer architectures with Kubernetes. Our stock Kubernetes architecture is as follows:

Architecture

MinIO provides a consistent, performant and scalable object store for any Kubernetes distribution. MinIO is Kubernetes-native by design and S3 compatible from inception. Developers can quickly deploy persistent object storage for all of their cloud native applications. The combination of MinIO and Kubernetes provides a powerful platform that allows applications to scale across any multi-cloud and hybrid cloud infrastructure and still be centrally managed and secured, avoiding public cloud lock-in.

MinIO Operator integrates natively with Kubernetes to provide:

Storage Classes and Tiering
Storage Classes and Tiering
Tier across NVMe, HDD and Public Cloud Storage.
External Load Balancing
External Load Balancing
Load balance incoming requests with NGINX ingress controller.
Encryption Key Management
Encryption Key Management
Manage encryption keys with HashiCorp Vault.
Identity Management
Identity Management
Manage identity and policy with OpenID Connect compatible Keycloak IDP.
Certificate Management
Certificate Management
Configure and manage certificates with Rancher Certificate Manager and Let's Encrypt.
Monitoring and Alerting
Monitoring and Alerting
Track Metrics and issue alerts using Rancher Monitoring or Grafana.
Logging and Auditing
Logging and Auditing
Output logs to an Elastic Stack for analysis.

Kubernetes Storage Classes and Tiering

The key requirement to deploy MinIO at scale is the ability to tier across Kubernetes storage classes (NVMe, HDD, Public Cloud). This allows enterprises to manage both cost and performance.

MinIO supports automatic transition of aged objects from the fast NVMe tier to a more cost-efficient HDD tier and even cost-optimized cold Public Cloud storage tiers.

When tiering, MinIO presents a unified namespace across the tiers. Movement across the tiers is transparent to the application and is triggered by customer policies.

MinIO and Kubernetes enable hybrid and multi-cloud storage safely and securely by encrypting objects at the source - ensuring customers retain total control over the data. Kubernetes efficiently manages data across persistent block storage and cheaper object storage tiers when deployed inside the public cloud.

External Load Balancing

All of MinIO’s communication is based on HTTPs, RESTFUL APIs and will support any standard, Kubernetes compatible ingress controller. This includes hardware based and software defined solutions. The most popular choice is NGINX.

Encryption Key Management

We recommend using HashiCorp Vault to store keys outside of the object storage system. This is a best practice for cloud native applications.

We recommend encryption be enabled by default on all buckets in production environments. MinIO uses AES-256-GCM or ChaCha20-Poly1305 encryption to protect data integrity and confidentiality with negligible performance impact.

MinIO supports all of the three server-side encryption (SSE-KMS, SSE-S3 and SSE-C) modes. SSE-S3 and SSE-KMS integrate with the KMS on the server side, whereas SSE-C uses the client supplied keys.MinIO supports setting a bucket-level default encryption key in the KMS with support for AWS-S3 semantics (SSE-S3). Clients also specify a separate key on the KMS using SSE-KMS request headers.

MinIO relies on an external KMS to bootstrap its internal key encryption server (KES service) to enable high-performance, per object encryption. Each tenant runs its own KES server in an isolated namespace.

Identity Management

Manage single sign-on (SSO) for Kubernetes and MinIO through a third party OpenID Connect/LDAP compatible identity provider, for example Keycloak, Okta/Auth0, Google, Facebook, ActiveDirectory and OpenLDAP. MinIO recommends OpenID Connect compatible Keycloak IDP.

Administrators can centrally manage user/application identity using an external IDP. MinIO enhances the IDP, providing AWS IAM-style users, groups, roles, policies and token service API. Enterprises gain significant architectural flexibility with an infrastructure independent and unified identity and access management (IAM) layer.

Certificate Management

TLS is used to encrypt all traffic, including internode traffic, between applications and MinIO. TLS certificates establish the identity of network-connected resources, such as a MinIO server domain, and secure network communications.

The MinIO Operator automatically configures, provisions, manages and updates certificates for MinIO tenants. The tenants are completely isolated from each other in their own Kubernetes namespace with their own certificates for improved security.

Monitoring and Alerting

MinIO recommends using Prometheus-compatible systems for monitoring and alerting when running on Kubernetes. MinIO publishes every object storage related Prometheus metric imaginable, from bucket capacity to access metrics. Those metrics can be collected and visualized in any Prometheus-compatible tool or the MinIO Console.

External monitoring solutions scrape the MinIO Prometheus endpoint at regular intervals. MinIO recommends Grafana to monitor the Prometheus feed in MinIO. Administrators establish baselines and set alert thresholds for notifications, which can then be routed via Alertmanager to a notification platform such as PagerDuty, Freshservice or even SNMP.

Logging and Auditing

Enabling MinIO auditing generates a log for every operation on the object storage cluster. In addition to the audit log, MinIO also logs console errors for operational troubleshooting purposes.

MinIO supports outputting logs to the Elastic Stack (or third parties) for analysis and alerting. To streamline operations, we recommend using the same logging and audit tool for Kubernetes and MinIO.

MinIO is Kubernetes Storage

Kubernetes relies on object storage. Learn the ins and outs of Kubernetes native object storage from the engineers who built MinIO.

MinIO Integrates with Top-Tier Kubernetes Services

Learn more

You are using Internet Explorer version 11 or lower. Due to security issues and lack of support for web standards, it is highly recommended that you upgrade to a modern browser.