Advertisement
PeachLemonade

s8t1

May 1st, 2024
111
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.79 KB | None | 0 0
  1. from pyspark.sql import SparkSession
  2.  
  3. spark_jars_packages = ",".join(
  4.         [
  5.             "org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0"
  6.         ]
  7.   )
  8.  
  9. spark = (
  10.     SparkSession.builder
  11.     .master("local")
  12.         .appName('test connect to kafka')
  13.     .config("spark.jars.packages", spark_jars_packages)
  14.     .getOrCreate()
  15. )
  16.  
  17. df = (spark.read
  18.                 .format('kafka')
  19.        .option('kafka.bootstrap.servers', 'rc1b-2erh7b35n4j4v869.mdb.yandexcloud.net:9091')
  20.        .option('kafka.security.protocol', 'SASL_SSL')
  21.        .option('kafka.sasl.mechanism', 'SCRAM-SHA-512')
  22.        .option('kafka.sasl.jaas.config','org.apache.kafka.common.security.scram.ScramLoginModule required username=\"de-student\" password=\"ltcneltyn\";')
  23.        .option("subscribe", "persist_topic")
  24.        .load())
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement