API tools faq
paste
Advertisement
PeachLemonade

s8t1

PeachLemonade
May 1st, 2024
111
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.79 KB | None | 0 0
raw download clone embed print report
  1. from pyspark.sql import SparkSession
  2.  
  3. spark_jars_packages = ",".join(
  4.         [
  5.             "org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.0"
  6.         ]
  7.   )
  8.  
  9. spark = (
  10.     SparkSession.builder
  11.     .master("local")
  12.         .appName('test connect to kafka')
  13.     .config("spark.jars.packages", spark_jars_packages)
  14.     .getOrCreate()
  15. )
  16.  
  17. df = (spark.read
  18.                 .format('kafka')
  19.        .option('kafka.bootstrap.servers', 'rc1b-2erh7b35n4j4v869.mdb.yandexcloud.net:9091')
  20.        .option('kafka.security.protocol', 'SASL_SSL')
  21.        .option('kafka.sasl.mechanism', 'SCRAM-SHA-512')
  22.        .option('kafka.sasl.jaas.config','org.apache.kafka.common.security.scram.ScramLoginModule required username=\"de-student\" password=\"ltcneltyn\";')
  23.        .option("subscribe", "persist_topic")
  24.        .load())
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!