Open Source, S3 Compatible, Enterprise Hardened and Really, Really Fast
At the heart of the hyper-scale revolution is the concept that compute and storage should be separate and best of breed. This architecture leads to better performance, capacity utilization and I/O while simultaneously lowering space, cooling and power requirements. MinIO is committed to building the best, private-cloud, software-defined object storage system on the planet.
With its focus on high performance, MinIO enables enterprises to support multiple use cases with the same platform. For example, MinIO’s performance characteristics mean that you can run multiple Spark, Presto, and Hive queries, or to quickly test, train and deploy AI algorithms, without suffering a storage bottleneck. MinIO object storage is used as the primary storage for cloud native applications that require higher throughput and lower latency than traditional object storage can provide.
MinIO leverages the hard won knowledge of the web scalers to bring a simple scaling model to object storage. This accompanies a strongly held belief that “simple things scale.” At MinIO scaling starts with a single cluster which can be federated with other MinIO clusters to create a global namespace, spanning multiple data centers if needed. Expansion of the namespace is possible by adding more clusters, more racks until the goal is achieved.
Minimalism is a guiding design principle at MinIO. Simplicity reduces opportunities for errors, improves uptime, delivers reliability while serving as the foundation for performance. MinIO can be installed and configured within minutes simply by downloading a single binary and then executing. The amount of configuration options and variations is kept to a minimum which results in near-zero system administration tasks and fewer paths to failures. Upgrading MinIO is done with a single command which is non-disruptive and incurs zero downtime - lowering total cost of ownership.
Amazon’s S3 API is the defacto standard in the object storage world and represents the most modern storage API in the market. MinIO adopted S3 compatibility early on and was the first to extend it to support S3 Select. MinIO prides itself on the comprehensive nature of its compatibility and points to the fact that more than 750 organizations use MinIO’s S3 Gateway - more than the rest of the industry combined.
MinIO is 100% open source under the Apache V2 license. This means that MinIO’s customers are free from lock in, free to inspect, free to innovate, free to modify and free to redistribute. Indeed, MinIO powers product offerings from multiple Fortune 500 organizations. The purity of MinIO’s licensing approach and the quality of its subscription support offering have made the company the fastest growing, private cloud object storage system in the world.
MinIO’s enterprise class features represent the standard in the object storage space. From the AWS S3 API to S3 Select and our implementations of inline erasure coding and security our code is widely admired and frequently copied by some of the biggest names in technology and business.
MinIO protects data with per-object, inline erasure coding which is written in assembly code to deliver the highest performance possible. MinIO uses Reed-Solomon code to stripe objects into n/2 data and n/2 parity blocks - although these can be configured to any desired redundancy level. This means that in a 12 drive setup, an object is sharded across as 6 data and 6 parity blocks. Even if you lose as many as 5 ((n/2)–1) drives, be it parity or data, you can still reconstruct the data reliably from the remaining drives. MinIO’s implementation ensures that objects can be read or new objects written even if multiple devices are lost or unavailable. Finally, MinIO's erasure code is at the object level and can heal one object at a time.
Silent data corruption or bitrot is a serious problem faced by disk drives resulting in data getting corrupted without the user’s knowledge. The reasons are manifold (aging drives, current spikes, bugs in disk firmware, phantom writes, misdirected reads/writes, driver errors, accidental overwrites) but the result is the same - compromised data.
MinIO’s optimized implementation of the HighwayHash algorithm ensures that it will never read corrupted data - it captures and heals corrupted objects on the fly. Integrity is ensured from end to end by computing hash on READ and verifying it on WRITE from the application, across the network and to the memory/drive. The implementation is designed for speed and can achieve hashing speeds over 10 GB/sec on a single core on Intel CPUs.
It is one thing to encrypt data in flight, it is another to protect data at rest. MinIO supports multiple, sophisticated server-side encryption schemes to protect data - wherever it may be. MinIO’s approach assures confidentiality, integrity and authenticity with negligible performance overhead. Server side and client side encryption are supported using AES-256-GCM, ChaCha20-Poly1305 and AES-CBC. Encrypted objects are tamper-proofed with AEAD server side encryption. Additionally, MinIO is compatible with and tested against all commonly used Key Management solutions (e.g. HashiCorp Vault).
MinIO uses a key-management-system (KMS) to support SSE-S3. If a client requests SSE-S3, or auto-encryption is enabled, the MinIO server encrypts each object with a unique object key which is protected by a master key managed by the KMS. Given the exceptionally low overhead, auto-encryption can be turned on for every application and instance.
When WORM is enabled, MinIO disables all APIs that can potentially mutate the object data and metadata. The means that data once written becomes tamper-proof. This has practical applications for a number of different regulatory requirements.
MinIO supports the most advanced standards in identity management, integrating with the OpenID connect compatible providers as well as key external IDP vendors. That means that access is centralized and passwords are temporary and rotated, not stored in config files and databases. Furthermore, access policies are fine grained and highly configurable which means that supporting multi-tenant and multi-instance deployments become simple.
The challenge with traditional replication approaches is that they do not scale effectively beyond a few hundred TB. Having said that, everyone needs a replication strategy to support disaster recovery and that strategy needs to span geographies, data centers and clouds. MinIO’s continuous replication is designed for large scale, cross data center deployments. By leveraging Lambda compute notifications and object metadata it can compute the delta efficiently and quickly.
Lambda notifications ensure that changes are propagated immediately as opposed to traditional batch mode. Continuous replication means that data loss will be kept to a bare minimum should a failure occur - even in the face of highly dynamic datasets. Finally, like all that MinIO does, continuous replication is multi-vendor, meaning that your backup location can be anything from NAS to the public cloud.
The modern enterprise has data everywhere. MinIO allows those various instances to be combined to form a unified global namespace. Specifically, up to 32 MinIO servers can be combined into a Distributed Mode set and multiple Distributed Mode sets can be combined into a MinIO Server Federation. Each MinIO Server Federation provides a unified admin and namespace.
A MinIO Federation Server supports an unlimited number of Distributed Mode sets.
The impact of this approach is that an object store can scale massively for large, geographically distributed enterprise while retaining the ability to accommodate a variety of applications (S3 Select, MinSQL, Spark, Hive, Presto, TensorFlow, H20) from a single console.
All enterprises are adopting a multi-cloud strategy. This also includes private clouds. As a result, your bare-metal virtualization containers and public cloud services (including non-S3 providers like Google, Microsoft and Alibaba) have to look identical. While the modern application is highly portable, the data that powers those applications is not.
Making that data available, wherever it may reside, is the primary challenge that MinIO addresses. MinIO runs on bare metal, network attached storage and every public cloud. More importantly, MinIO ensures your view of that data looks exactly the same from an application and management perspective via the Amazon S3 API.
MinIO, can go even further, making your existing storage infrastructure compatible with Amazon S3. The implications are profound. Now organizations can truly unify their data infrastructure - from file to block, all appearing as objects accessible via the Amazon S3 API without the requirement for migration.
MinIO is designed to be cloud native and can run as lightweight containers managed by external orchestration services such as Kubernetes. The entire server is a ~40MB static binary and is highly efficient in its use of CPU and memory resources - even under high loads. The result is that you can co-host a large number of tenants on shared hardware.
MinIO operates on commodity servers with locally attached drives (JBOD/JBOF). All of the servers in a cluster are equal in capability (fully symmetrical architecture). There are no name nodes or metadata servers.
MinIO writes data and metadata together as objects, eliminating the need for a metadata database. In addition MinIO performs all functions (erasure code, bitrot check, encryption) as inline, strictly consistent operations. The result is that MinIO is exceptionally resilient.
Each MinIO cluster is a collection of distributed MinIO servers with one process per node. MinIO runs in the user space as a single process and uses lightweight co-routines for high concurrency. Drives are grouped into erasure sets (16 drives per set by default) and objects are placed on these sets using a deterministic hashing algorithm.
MinIO is designed for large scale, multi-data center cloud storage services. Each tenant runs their own MinIO cluster, fully isolated from other tenants giving them the ability to protect them from any disruption on upgrade, update, security incidents. Each tenant scales independently by federating clusters across geographies.
While MinIO is hardware agnostic, these industry standard, widely available boxes are tested to deliver against the exceptional capabilities of our software. We work tirelessly to optimize our software against the latest hardware and, as a result, can significantly outperform overpriced, outdated appliances.