Blog
Using MinIO as external tables to extend Snowflake
Query and analyze multiple data sources residing on MinIO with the Snowflake Data Cloud. Now enterprises can access Snowflake databases no matter where they exist - in the public cloud, the private cloud, even set up streaming edge instances. This includes AWS, GCP, Azure, Tanzu, OpenShift, HPE Ezmeral, SUSE Rancher and of course traditional bare metal deployments.
Using external tables, enterprises can enjoy the full capabilities of Snowflake without incurring the cost of moving data into the Snowflake Data Cloud. It also enables enterprises to take advantage of multi-tenant data or perform analysis on data that cannot be moved for compliance or other business reasons.
Just query and go using the standard Snowflake CLI (SnowSQL) commands.
With any economic or technical constraints removed, the enterprise can unleash the Snowflake Data Cloud on all of its data - no matter where it resides. Further, with MinIO’s industry leading performance characteristics, queries can run against massive, multi-petabyte data stores and complete quickly.
This means better utilization for both Snowflake and the Enterprise’s data.
To guarantee that the right data is available to the right user, it is imperative that there be fine-grained access control on these multi-cloud data lakes.
For authorization, MinIO’s ability to integrate with 3rd party IDPs ensures that access to the object store is restricted to only those who need it. To authenticate that those users only have access to the specific resources that they need, MinIO’s sophisticated Policy Based Access Control (PBAC) capabilities ensure that this is not an after-thought.
Snowflake is a mission critical application in the enterprise. That means the Snowflake architecture needs to be built for resilience and high availability. That resiliency cannot just account for region failure in the cloud, it needs to account for the failure of the entire cloud.
MinIO’s active-active, multi-site, strictly consistent replication provides the framework to sustain full cloud failure and Snowflake’s ability to run in GCP, Azure and AWS provides similar functionality.
With the addition of the external tables functionality, now enterprises can set up streaming pipelines that persist the data on MinIO - in the cloud or on prem. Snowflake can be configured to execute queries on this data in real time - adding a critical new dimension to the Snowflake experience.
MinIO’s integration with Snowflake is extremely simple. The MinIO admin sets up the bucket that is meant to be available for use with Snowflake and provides the end-points and the credentials needed.
The Snowflake admin simply makes the standard “create stage” and “create external table” SnowSQL commands with the provided end-points and credentials and the integration is complete. From then on, all Snowflake commands that are applicable to external tables are available to the Snowflake user.
External tables functionality was introduced in the mid-June 2022 release.
Object stores are often the core infrastructure that data intensive applications depend on. Because of MinIO’s performance characteristics it is used across a number of applications and use cases.
Pretty much any Snowflake operations that work with external tables should work with the MinIO integration. This includes copy, select, join, etc. Please note that external tables are read-only so no update or delete operations can be performed on them.
MinIO can run on almost all hardware profiles. Our recommended partners are listed here ↗.
MinIO natively supports multi-tenancy.
Separate buckets can be set up in MinIO for different applications. The Snowflake administrator can make these buckets available as separate external tables for different applications and/or users.
MinIO’s encryption and Snowflake’s encryption are completely independent. While setting up the integration, Snowflake passes both the credentials and the encryption key to MinIO. From then on, data fetched by Snowflake from an encrypted bucket in MinIO will be decrypted automatically. This data is encrypted on the wire during transport using TLS 1.2+
MinIO’s server-side replication feature supports both active-passive and active-active configurations to keep two data-centers in sync continuously. Objects are replicated immediately as long as there is sufficient bandwidth.
Snowflake is a proprietary license. Visit Snowflake’s pricing page ↗ for more information. MinIO is licensed under GNU AGPL v3. Pricing for commercial licenses of MinIO can be found here ↗.
MinIO’s white glove, direct-to-engineer model will ensure that this use case is fully supported for commercial customers.
