MinIO for Microsoft Azure Kubernetes Service

Abstract watercolor painting with dynamic red and pink brush strokes

Customers run MinIO on Microsoft Azure Kubernetes Service (AKS) for three reasons.

MinIO serves as the consistency layer in a hybrid cloud or multi-cloud deployment, bringing S3 API compatibility to Azure and enabling AI/ML workload portability.

MinIO is Kubernetes native and high performance it can deliver predictable performance across public, private and edge cloud environments.

Running MinIO on AKS provides control over the software stack with flexibility to avoid cloud lock-in.

Architecture

AKS is a highly available, secure and fully managed Kubernetes service from Microsoft Azure. Azure Kubernetes Service (AKS) offers serverless Kubernetes, an integrated continuous integration and continuous delivery (CI/CD) experience, and enterprise-grade security and governance.

MinIO provides a portable high-performance object storage system across all of the major Kubernetes platforms (AWS, Tanzu, GCP, OpenShift). Developers can easily get an Amazon S3 compatible persistent storage service for all of their cloud native applications running on AKS. Unlike AWS S3, MinIO enables the applications to scale across any multi-cloud and hybrid cloud infrastructure, without requiring expensive software rewrites or proprietary integrations.

AIStor Operator integrates natively with AKS and Azure features to provide:

Storage Classes and Tiering‍

Tier across Azure SSD Managed Disks (Azure CSI Volumes), Azure BlobStore and Azure Cool Blob.

A key requirement for deploying MinIO at scale on AKS and Azure is the ability to transition objects across Azure cloud storage types. Specifically, you want support for cost-optimized "Hot-Warm" and "Hot-Cold" deployment topologies.

MinIO can use an Azure blob as a remote tier for automatic transition of aged objects based on user-configured rules. For example, you can create one rule that transitions objects to a tier with the Azure "Hot" access tier, while another rule transitions objects to a tier with the Azure "Cold" access tier.

MinIO supports transitioning an object once - so while you cannot configure "waterfall" or "chain" transition from MinIO -> "Hot" -> "Cold", you can configure multiple rules per bucket using prefix and object tags to apply granular transition behavior to the preferred remote tier. MinIO's only requirement is that the remote storage must support immediate retrieval of objects - no rehydration, latency, or wait times.

MinIO tiering requires no client-side logic changes. Your clients can continue to request an object through MinIO, and MinIO handles retrieving the object from Azure and returning it transparently. MinIO also supports using the S3 restore API for returning objects back to the "hot" MinIO deployment.

MinIO's tiering capability extends to hybrid cloud environments where the MinIO JBOD/JBOF deployment acts as the performance-optimized "hot" tier on the private cloud, while Azure provides cost-optimized "warm" and "cold" tiers. Leverage MinIO TLS and Server-Side Encryption to further protect all data in both clouds, at rest and in flight.

External Load Balancing

Load balance incoming requests with Azure Load Balancer.

AKS offers built-in Azure Load Balancer to provide automatic load balancing and routing services across multiple MinIO tenants for applications accessing the storage service from outside of AKS. Exposing a MinIO tenant to external traffic can be done by simply adding annotations to a MinIO tenant.

Encryption Key Management

Load balance incoming requests with Azure Load Balancer.

For cloud-native applications, the best practices dictate storing keys outside of the object system in an external vault. Azure Key Vault (AKV) is a secure and resilient service for managing keys by API across Azure services.

For those with more stringent security requirements or for consistency purposes, MinIO integrates with a number of external Key Management Services that operate outside of Azure.

When using MinIO as the object storage in a public instance of AKS, encryption on-disk is strongly recommended. MinIO uses AES-256-GCM or ChaCha20-Poly1305 encryption to protect data integrity and confidentiality without impacting performance. The AIStor Operator allows for tenants to be configured for the Azure Key Vault or a supported third-party KMS for automatic server-side encryption of objects.

MinIO supports setting a bucket-level default encryption key in the KMS with support for per-object encryption keys (SSE-KMS) and a broad per-deployment encryption key (SSE-S3).

MinIO will use this KMS to bootstrap its internal key encryption server (KES service) to enable high-performance per object encryption. Each tenant runs its own KES server in an isolated namespace.

Identity Management

Manage identity and policy with Azure ActiveDirectory.

When running MinIO on AKS, customers can manage single sign-on (SSO) through Azure ActiveDirectory or third party OpenID Connect/LDAP compatible identity providers like Okta/Auth0, Google, Facebook, Keycloak and OpenLDAP.

An external IDP allows administrators to centrally manage user/application identity. MinIO builds on top of the IDP, providing AWS IAM-style users, groups, roles, policies and token service API. The ability to have a unified identity and access management (IAM) layer independent of the infrastructure provides significant architectural flexibility

Certificate Management

Configure and manage certificates with JetStack Certificate Manager and Let’s Encrypt.

All traffic from the application to MinIO, including internode traffic, is encrypted with TLS. TLS certificates are used to secure network communications and establish the identity of network-connected resources, such as a MinIO Server.

AKS uses Jetstack cert-manager to automatically generate and configure Let’s Encrypt certificates. MinIO integrates with the Jetstack cert-manager so you can use the AIStor Operator to configure, provision, manage and update certificates for the MinIO tenants. The tenants are completely isolated from each other in their own Kubernetes namespace with their own certificates for improved security.

Monitoring and Alerting

Track metrics and issue alerts using Azure Monitor.

MinIO recommends using Azure Monitor as a Prometheus-compatible system for monitoring and alerting when deploying MinIO on AKS. MinIO publishes every object storage related Prometheus metric imaginable, from bucket capacity to access metrics. Those metrics can be collected and visualized in Azure Monitor, Grafana, the MinIO Console or any Prometheus-compatible tool already being used for Azure Kubernetes Service monitoring.

These same tools can also be used to establish baselines and set alert thresholds for notifications, which can then be routed to a notification platform such as PagerDuty, Freshservice or even SNMP.

Logging and Auditing

Output logs to Azure Monitor for analysis.

Enabling MinIO auditing generates a log for every operation on the object storage cluster. In addition to the audit log, MinIO also logs console errors for operational troubleshooting purposes.

MinIO supports sending logs to the Azure Monitor and using the Log Analytics features.