Product
Overview
What is AIStor?
Features
Replication Replication Console Console Cache Cache Encryption Encryption Versioning Versioning Observability Observability Object Immutability Object Immutability Key Management Server Key Management Server S3 Compatibility S3 Compatibility Identity + Access Mgt Identity + Access Mgt Catalog Catalog Object Prompt Object Prompt Lifecycle Lifecycle Firewall Firewall AIHub AIHub
Tools
Erasure Code Calculator Erasure Code Calculator Determine your raw and usable capacity across a range of erasure coding settings. Reference Hardware Reference Hardware MinIO’s recommended Configuration and reference hardware for building large scale data infrastructure.
Solutions
AI Storage Learn how MinIO is leading the AI storage market from its exclusive features to performance at scale. HDFS Migration Modernize and simplify your big data storage infrastructure with cloud native storage with AIStor. Snowflake Learn how to leverage Snowflake external tables to query data without having to move it.
Integrations Integrations with MinIO Browse our expansive list of
integrations with links to the relevant
documentation.
Veeam Learn how MinIO and Veeam have partnered deliver superior RTO and RPO. Modern Datalakes Learn how modern, multi-engine data lakeshouses depend on MinIO's AIStor. Equinix Repatriate your data onto the cloud you control with MinIO and Equinix. SQL Server Learn how to leverage SQL Server 2022 with MinIO to run queries on your data without having to move it. Commvault Learn how Commvault and MinIO are partnered to deliver performance at scale for mission critical backup and restore workloads. Hybrid Cloud Learn how enterprises use MinIO to build AI data infrastructure that runs on any cloud - public, private or colo. Splunk Find out how MinIO is delivering performance at scale for Splunk SmartStores. VMware Discover how MinIO integrates with VMware across the portfolio from the Persistent Data platform to TKG and how we support their Kubernetes ambitions.
Open Source
Github Community Explore, experiment, ask questions and contribute. Docs Development guides. Slack Channel Open Forum for discussing topics related to MinIO.
Download MinIO Object Store Get MinIO's open source server, client, and SDK.
Resources
Resources Home Browse our library of white papers, solution briefs, benchmarks and videos. Blog Product updates, company news, and educational content. Training Learn the MinIO Way and Become a Data Infrastructure Expert.
Events Join us to take advantage to meet with MinIO partners, employees, and subject-matter experts.
Partner Pricing Login Download

Documentation

MinIO Object Storage for Linux

Kafka Notification Settings

This page documents settings for configuring an Kafka service as a target for Bucket Notifications. See Publish Events to Kafka for a tutorial on using these settings.

You can establish or modify settings by defining:

  • an environment variable on the host system prior to starting or restarting the MinIO Server. Refer to your operating system’s documentation for how to define an environment variable.

  • a configuration setting using mc admin config set.

  • a configuration setting using the MinIO Console’s Administrator > Settings pages.

If you define both an environment variable and the similar configuration setting, MinIO uses the environment variable value.

Some settings have only an environment variable or a configuration setting, but not both.

Important

Each configuration setting controls fundamental MinIO behavior and functionality. MinIO strongly recommends testing configuration changes in a lower environment, such as DEV or QA, before applying to production.

Multiple Kafka Targets

You can specify multiple Kafka service endpoints by appending a unique identifier _ID for each set of related Kafka settings on to the top level key.

Examples

For example, the following commands set two distinct Kafka service endpoints as PRIMARY and SECONDARY respectively:

export MINIO_NOTIFY_KAFKA_ENABLE_PRIMARY="on"
export MINIO_NOTIFY_KAFKA_BROKERS_PRIMARY="https://kafka1.example.net:9200, https://kafka2.example.net:9200"

export MINIO_NOTIFY_KAFKA_ENABLE_SECONDARY="on"
export MINIO_NOTIFY_KAFKA_BROKERS_SECONDARY="https://kafka1.example.net:9200, https://kafka2.example.net:9200"

mc admin config set notify_kafka:primary \
   brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
   [ARGUMENT=VALUE ...]

mc admin config set notify_kafka:secondary \
   brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
   [ARGUMENT=VALUE ...]

Notice that for configuration settings, the unique identifier appends to notify_kafka only, not to each individual argument.

Settings

Enable

Required

Specify on to enable publishing bucket notifications to a Kafka service endpoint.

Defaults to off.

The top-level configuration key for defining an Kafka service endpoint for use with MinIO bucket notifications.

Use mc admin config set to set or update an Kafka service endpoint. The brokers argument is required for each target. Specify additional optional arguments as a whitespace (" ")-delimited list.

mc admin config set notify_kafka \
  brokers="https://kafka1.example.net:9200, https://kafka2.example.net:9200"
  [ARGUMENT="VALUE"] ... \

Brokers

Required

Specify a comma-separated list of Kafka broker addresses. For example:

"kafka1.example.com:2021,kafka2.example.com:2021"

Changed in version RELEASE.2023-05-27T05-56-19Z: MinIO checks the health of the specified URL (if it is resolvable and reachable) prior to adding the target. MinIO no longer blocks adding new notification targets if existing targets are offline.

Topic

Optional

Specify the name of the Kafka topic to which MinIO publishes bucket events.

SASL

Optional

Specify on to enable SASL authentication.

SASL Username

Optional

Specify the username for performing SASL/PLAIN or SASL/SCRAM authentication to the Kafka broker(s).

SASL Password

Optional

Specify the password for performing SASL/PLAIN or SASL/SCRAM authentication to the Kafka broker(s).

Changed in version RELEASE.2023-06-23T20-26-00Z: MinIO redacts this value when returned as part of mc admin config get.

SASL Mechanism

Optional

Specify the SASL mechanism to use for authenticating to the Kafka broker(s). MinIO supports the following mechanisms:

  • PLAIN (Default)

  • SHA256

  • SHA512

TLS Client Auth

Optional

Specify the client authentication type of the Kafka broker(s). The following table lists the supported values and their mappings

Value

Authentication Type

0

NoClientCert

1

RequestClientCert

2

RequireAnyClientCert

3

VerifyClientCertIfGiven

4

RequireAndVerifyClientCert

See ClientAuthType for more information on each client auth type.

TLS

Optional

Specify on to enable TLS connectivity to the Kafka broker(s).

TLS Skip Verify

Optional

Enables or disables TLS verification of the NATS service endpoint TLS certificates.

  • Specify on to disable TLS verification (Default).

  • Specify off to enable TLS verification.

Client TLS Cert

Optional

Specify the path to the client certificate to use for performing mTLS authentication to the Kafka broker(s).

Client TLS Key

Optional

Specify the path to the client private key to use for performing mTLS authentication to the Kafka broker(s).

Version

Optional

Specify the version of the Kafka cluster to assume when performing operations against that cluster. See the sarama reference documentation for more information on this field’s behavior.

Batch Size

Optional

Specify the integer value to use as the batch size for sending records to Kafka.

Changed in version RELEASE.2023-12-02T10-51-33Z: MinIO previously limited this value to 100.

Queue Directory

Optional

Specify the directory path to enable MinIO’s persistent event store for undelivered messages, such as /opt/minio/events.

MinIO stores undelivered events in the specified store while the Kafka server/broker is offline and replays the stored events when connectivity resumes.

Queue Limit

Optional

Specify the maximum limit for undelivered messages. Defaults to 100000.

Comment

Optional

Specify a comment to associate with the Kafka configuration.

Compression Codec

New in version MinIO: Server RELEASE.2023-12-09T18-17-51Z

Optional

Specify the compression codec to use when sending records to Kafka.

Supports the following values:

  • none

  • snappy

  • gzip

  • lz4

  • zstd

Compression Level

New in version MinIO: Server RELEASE.2023-12-09T18-17-51Z

Optional

Controls the level of compression applied based on the configured compression codec.

Specify an integer value greater than or equal to 0. The effect of the value depends on the selected codec.

This work is licensed under a Creative Commons Attribution 4.0 International License. 2020-Present, MinIO, Inc. Creative Commons License