ProductPromotion
Logo

Elixir

made by https://0x3d.site

GitHub - BigThinkcode/ex_aws_msk_iam_auth: AWS_MSK_IAM Authentication Plugin for Broadway Kafka
AWS_MSK_IAM Authentication Plugin for Broadway Kafka - BigThinkcode/ex_aws_msk_iam_auth
Visit Site

GitHub - BigThinkcode/ex_aws_msk_iam_auth: AWS_MSK_IAM Authentication Plugin for Broadway Kafka

GitHub - BigThinkcode/ex_aws_msk_iam_auth: AWS_MSK_IAM Authentication Plugin for Broadway Kafka

ex_aws_msk_iam_auth

ex_aws_msk_iam_auth is an authentication plugin for broadway_kafka. It enables Broadway Kafka clients to authenticate with Amazon's Managed Streaming for Apache Kafka(Amazon MSK) via AWS_MSK_IAM SASL mechanism.

Installation

Add the following dependency to your mix.exs

def deps do
  [
    {:ex_aws_msk_iam_auth, git: "https://github.com/BigThinkcode/ex_aws_msk_iam_auth"}
  ]
end

Usage

Broadway Kafka supports connecting to Kafka broker via SASL authentication. The following sample configuration shows how ex_aws_msk_iam_auth plugin can be used with it.

Ref: https://hexdocs.pm/broadway_kafka/BroadwayKafka.Producer.html#module-client-config-options

  client_config: [
            sasl:
              {
                :callback, 
                ExAwsMskIamAuth, 
                {:AWS_MSK_IAM, "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"}
              },
            ssl: true
          ]

Background

Broadway Kafka and brod

Broadway Kafka is a Kafka Connector for Broadway - an Elixir library to build concurrent, multi-stage data ingestion/processing pipelines with Elixir. Broadway Kafka is an amalgamation of awesome features from Broadway with Kafka as a producer. Internally, it uses brod as its Kafka client acting as a wrapper. Brod supports SASL PLAIN, SCRAM-SHA-256 and SCRAM-SHA-512 authentication mechanisms out of the box and also offers extension points to support custom authentication plugins.

AWS MSK Authentication Mechanisms

MSK supports two variants - MSK Fully Managed and MSK Serverless. In both the variants, Kafka service can be protected via SASL, in particular, AWS's custom SASL mechanism AWS_MSK_IAM(https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html). At the time of writing this library, MSK's Serverless variant's only supported authentication was AWS_MSK_IAM SASL mechanism.

Solution Diagram

Solution Diagram

Implementation

This library takes inspiration from its Java counterpart aws-msk-iam-auth

Relevant Issues/PRs

  1. https://github.com/dashbitco/broadway_kafka/issues/82
  2. https://github.com/dashbitco/broadway_kafka/pull/85
  3. https://github.com/aws-beam/aws_signature/issues/14

Articles
to learn more about the elixir concepts.

Resources
which are currently available to browse on.

mail [email protected] to add your project or resources here 🔥.

FAQ's
to know more about the topic.

mail [email protected] to add your project or resources here 🔥.

Queries
or most google FAQ's about Elixir.

mail [email protected] to add more queries here 🔍.

More Sites
to check out once you're finished browsing here.

0x3d
https://www.0x3d.site/
0x3d is designed for aggregating information.
NodeJS
https://nodejs.0x3d.site/
NodeJS Online Directory
Cross Platform
https://cross-platform.0x3d.site/
Cross Platform Online Directory
Open Source
https://open-source.0x3d.site/
Open Source Online Directory
Analytics
https://analytics.0x3d.site/
Analytics Online Directory
JavaScript
https://javascript.0x3d.site/
JavaScript Online Directory
GoLang
https://golang.0x3d.site/
GoLang Online Directory
Python
https://python.0x3d.site/
Python Online Directory
Swift
https://swift.0x3d.site/
Swift Online Directory
Rust
https://rust.0x3d.site/
Rust Online Directory
Scala
https://scala.0x3d.site/
Scala Online Directory
Ruby
https://ruby.0x3d.site/
Ruby Online Directory
Clojure
https://clojure.0x3d.site/
Clojure Online Directory
Elixir
https://elixir.0x3d.site/
Elixir Online Directory
Elm
https://elm.0x3d.site/
Elm Online Directory
Lua
https://lua.0x3d.site/
Lua Online Directory
C Programming
https://c-programming.0x3d.site/
C Programming Online Directory
C++ Programming
https://cpp-programming.0x3d.site/
C++ Programming Online Directory
R Programming
https://r-programming.0x3d.site/
R Programming Online Directory
Perl
https://perl.0x3d.site/
Perl Online Directory
Java
https://java.0x3d.site/
Java Online Directory
Kotlin
https://kotlin.0x3d.site/
Kotlin Online Directory
PHP
https://php.0x3d.site/
PHP Online Directory
React JS
https://react.0x3d.site/
React JS Online Directory
Angular
https://angular.0x3d.site/
Angular JS Online Directory