Timeline for kafka stream with AWS Glue - authorization issue
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 24, 2023 at 8:24 | comment | added | Smaillns | df.selectExpr( "CAST(id AS STRING) AS key", "to_json(struct(metadata,payload)) AS value" ).write.format("kafka").option("topic", topic).option( "kafka.ssl.endpoint.identification.algorithm", "https" ).option("kafka.bootstrap.servers", bootstrap_server) .option( "kafka.sasl.jaas.config", f'org.apache.kafka.common.security.plain.PlainLoginModule required username="{username}" password="{password}";', ).option( "kafka.sasl.mechanism", "PLAIN" ).option( "kafka.security.protocol", "SASL_SSL" ).mode("append").save() | |
| Aug 24, 2023 at 8:22 | comment | added | Smaillns | @NadimYounes you need a valid connection between your aws accnt and kafka cluster, for the kafka config you can set it using pyspark as the following | |
| Aug 14, 2022 at 11:12 | comment | added | Smaillns | But we need to add the JAAS config to the connection_options ! | |
| Aug 13, 2022 at 2:17 | comment | added | Prabhakar Reddy | The issue is here is not with Authentication and related to Authorization which indicates the credentials you are using doesn't have access to topic that you are trying to access. | |
| Aug 12, 2022 at 19:31 | history | edited | OneCricketeer |
edited tags; edited tags
|
|
| Aug 12, 2022 at 15:23 | history | edited | Smaillns | CC BY-SA 4.0 |
deleted 48 characters in body
|
| Aug 12, 2022 at 15:12 | history | asked | Smaillns | CC BY-SA 4.0 |