1

I want to stream a kafka topic from Glue job but I got the following error

StreamingQueryException: Not authorized to access topics: [topic_name]

This my current script

# Script generated for node Kafka Stream
dataframe_KafkaStream_node1 = glueContext.create_data_frame.from_options(
    connection_type="kafka",
    connection_options={
        "connectionName": "corp_cli_kafka_connection",
        "classification": "json",
        "startingOffsets": "latest",
        "topicName": "topic_name",
        "inferSchema": "true",
        "typeOfData": "kafka"
    },
    transformation_ctx="dataframe_KafkaStream_node1",
)

How to set the SASL JAAS config ?

"kafka.bootstrap.servers": "server_url:9092",
"kafka.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=XXXXXXXXXX password=XXXXXXXXXXXX;",
"kafka.sasl.mechanism": "PLAIN",
"kafka.security.protocol": "SASL_SSL"
Improve this question
4
  • The issue is here is not with Authentication and related to Authorization which indicates the credentials you are using doesn't have access to topic that you are trying to access.
    – Prabhakar Reddy
    Commented Aug 13, 2022 at 2:17
  • But we need to add the JAAS config to the connection_options !
    – Smaillns
    Commented Aug 14, 2022 at 11:12
  • @NadimYounes you need a valid connection between your aws accnt and kafka cluster, for the kafka config you can set it using pyspark as the following
    – Smaillns
    Commented Aug 24, 2023 at 8:22
  • df.selectExpr( "CAST(id AS STRING) AS key", "to_json(struct(metadata,payload)) AS value" ).write.format("kafka").option("topic", topic).option( "kafka.ssl.endpoint.identification.algorithm", "https" ).option("kafka.bootstrap.servers", bootstrap_server) .option( "kafka.sasl.jaas.config", f'org.apache.kafka.common.security.plain.PlainLoginModule required username="{username}" password="{password}";', ).option( "kafka.sasl.mechanism", "PLAIN" ).option( "kafka.security.protocol", "SASL_SSL" ).mode("append").save()
    – Smaillns
    Commented Aug 24, 2023 at 8:24
Related questions
6
Integrating Apache Kafka with Apache Spark Streaming using Python
7
Kafka 10 - Python Client with Authentication and Authorization
1
Issue using Kafka with Spark using pyspark
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.