Apache Spark Introduction

6 thoughts on “Apache Spark Introduction”

1. Hello,

   Can you please help me know why I am facing following issue:

   I am trying to print text from the twitter post. I am able to print the text when producer sends only the text from the tweet to Spark i.e. producer.send( topic, tweet [“text”])

   However, on sending complete JSON object, that is:
   json_data = json.dumps(tweet)
   producer.send (topic, json_data)
   nothing prints on the console using pprint() function.

   Spark File has:

   os.environ[“SPARK_HOME”] = “C:spark”
   sc = SparkContext(appName=”PythonStreamingDirectKafkaWordCount”)
   ssc = StreamingContext(sc, 5)
   sc.setLogLevel(“ERROR”)
   brokers, topic = sys.argv[1:]
   print (“creating Kafka Direct Stream Object”)
   kvs = KafkaUtils.createDirectStream(ssc, [topic], {“metadata.broker.list”: brokers})
   print (“Json object extracted from kvs map”)
   tweetRDD = kvs.map(lambda (k, v): json.loads(v)).map(lambda tweet: tweet[“text”])
   print(“TweetRDD Created”)
   tweetRDD.pprint()
   print(“Should Print TweetRDD”)
   ssc.start()
   ssc.awaitTermination()

   Producer File:
   import tweepy
   from kafka import KafkaProducer
   import json

   class StdOutListener(tweepy.StreamListener):

   def on_status(self, tweet):
   try:
   print tweet.text
   json_data = json.dumps(tweet)
   producer.send(mytopic, json_data)
   ………
   if __name__ == ‘__main__’:
   producer = KafkaProducer(bootstrap_servers=”localhost:9092″, value_serializer=lambda v: json.dumps(v).encode(‘utf-8’))

   Reply

2. Hello,

   Can you please help me know why I am facing following issue:

   I am trying to print text from the twitter post. I am able to print the text when producer sends only the text from the tweet to Spark i.e. producer.send( topic, tweet [“text”])

   However, on sending complete JSON object, that is:
   json_data = json.dumps(tweet)
   producer.send (topic, json_data)
   nothing prints on the console using pprint() function.

   Spark File has:

   os.environ[“SPARK_HOME”] = “C:spark”
   sc = SparkContext(appName=”PythonStreamingDirectKafkaWordCount”)
   ssc = StreamingContext(sc, 5)
   sc.setLogLevel(“ERROR”)
   brokers, topic = sys.argv[1:]
   print (“creating Kafka Direct Stream Object”)
   kvs = KafkaUtils.createDirectStream(ssc, [topic], {“metadata.broker.list”: brokers})
   print (“Json object extracted from kvs map”)
   tweetRDD = kvs.map(lambda (k, v): json.loads(v)).map(lambda tweet: tweet[“text”])
   print(“TweetRDD Created”)
   tweetRDD.pprint()
   print(“Should Print TweetRDD”)
   ssc.start()
   ssc.awaitTermination()

   Producer File:
   import tweepy
   from kafka import KafkaProducer
   import json

   class StdOutListener(tweepy.StreamListener):

   def on_status(self, tweet):
   try:
   print tweet.text
   json_data = json.dumps(tweet)
   producer.send(mytopic, json_data)
   ………
   if __name__ == ‘__main__’:
   producer = KafkaProducer(bootstrap_servers=”localhost:9092″, value_serializer=lambda v: json.dumps(v).encode(‘utf-8’))

   Reply

3. Hello,

   Can you please help me know why I am facing following issue:

   I am trying to print text from the twitter post. I am able to print the text when producer sends only the text from the tweet to Spark i.e. producer.send( topic, tweet [“text”])

   However, on sending complete JSON object, that is:
   json_data = json.dumps(tweet)
   producer.send (topic, json_data)
   nothing prints on the console using pprint() function.

   Spark File has:

   os.environ[“SPARK_HOME”] = “C:spark”
   sc = SparkContext(appName=”PythonStreamingDirectKafkaWordCount”)
   ssc = StreamingContext(sc, 5)
   sc.setLogLevel(“ERROR”)
   brokers, topic = sys.argv[1:]
   print (“creating Kafka Direct Stream Object”)
   kvs = KafkaUtils.createDirectStream(ssc, [topic], {“metadata.broker.list”: brokers})
   print (“Json object extracted from kvs map”)
   tweetRDD = kvs.map(lambda (k, v): json.loads(v)).map(lambda tweet: tweet[“text”])
   print(“TweetRDD Created”)
   tweetRDD.pprint()
   print(“Should Print TweetRDD”)
   ssc.start()
   ssc.awaitTermination()

   Producer File:
   import tweepy
   from kafka import KafkaProducer
   import json

   class StdOutListener(tweepy.StreamListener):

   def on_status(self, tweet):
   try:
   print tweet.text
   json_data = json.dumps(tweet)
   producer.send(mytopic, json_data)
   ………
   if __name__ == ‘__main__’:
   producer = KafkaProducer(bootstrap_servers=”localhost:9092″, value_serializer=lambda v: json.dumps(v).encode(‘utf-8’))

   Reply

4. Hello,

   Can you please help me know why I am facing following issue:

   I am trying to print text from the twitter post. I am able to print the text when producer sends only the text from the tweet to Spark i.e. producer.send( topic, tweet [“text”])

   However, on sending complete JSON object, that is:
   json_data = json.dumps(tweet)
   producer.send (topic, json_data)
   nothing prints on the console using pprint() function.

   Spark File has:

   os.environ[“SPARK_HOME”] = “C:spark”
   sc = SparkContext(appName=”PythonStreamingDirectKafkaWordCount”)
   ssc = StreamingContext(sc, 5)
   sc.setLogLevel(“ERROR”)
   brokers, topic = sys.argv[1:]
   print (“creating Kafka Direct Stream Object”)
   kvs = KafkaUtils.createDirectStream(ssc, [topic], {“metadata.broker.list”: brokers})
   print (“Json object extracted from kvs map”)
   tweetRDD = kvs.map(lambda (k, v): json.loads(v)).map(lambda tweet: tweet[“text”])
   print(“TweetRDD Created”)
   tweetRDD.pprint()
   print(“Should Print TweetRDD”)
   ssc.start()
   ssc.awaitTermination()

   Producer File:
   import tweepy
   from kafka import KafkaProducer
   import json

   class StdOutListener(tweepy.StreamListener):

   def on_status(self, tweet):
   try:
   print tweet.text
   json_data = json.dumps(tweet)
   producer.send(mytopic, json_data)
   ………
   if __name__ == ‘__main__’:
   producer = KafkaProducer(bootstrap_servers=”localhost:9092″, value_serializer=lambda v: json.dumps(v).encode(‘utf-8’))

   Reply

5. Hello,

   Can you please help me know why I am facing following issue:

   I am trying to print text from the twitter post. I am able to print the text when producer sends only the text from the tweet to Spark i.e. producer.send( topic, tweet [“text”])

   However, on sending complete JSON object, that is:
   json_data = json.dumps(tweet)
   producer.send (topic, json_data)
   nothing prints on the console using pprint() function.

   Spark File has:

   os.environ[“SPARK_HOME”] = “C:spark”
   sc = SparkContext(appName=”PythonStreamingDirectKafkaWordCount”)
   ssc = StreamingContext(sc, 5)
   sc.setLogLevel(“ERROR”)
   brokers, topic = sys.argv[1:]
   print (“creating Kafka Direct Stream Object”)
   kvs = KafkaUtils.createDirectStream(ssc, [topic], {“metadata.broker.list”: brokers})
   print (“Json object extracted from kvs map”)
   tweetRDD = kvs.map(lambda (k, v): json.loads(v)).map(lambda tweet: tweet[“text”])
   print(“TweetRDD Created”)
   tweetRDD.pprint()
   print(“Should Print TweetRDD”)
   ssc.start()
   ssc.awaitTermination()

   Producer File:
   import tweepy
   from kafka import KafkaProducer
   import json

   class StdOutListener(tweepy.StreamListener):

   def on_status(self, tweet):
   try:
   print tweet.text
   json_data = json.dumps(tweet)
   producer.send(mytopic, json_data)
   ………
   if __name__ == ‘__main__’:
   producer = KafkaProducer(bootstrap_servers=”localhost:9092″, value_serializer=lambda v: json.dumps(v).encode(‘utf-8’))

   Reply

6. Hello,

   Can you please help me know why I am facing following issue:

   I am trying to print text from the twitter post. I am able to print the text when producer sends only the text from the tweet to Spark i.e. producer.send( topic, tweet [“text”])

   However, on sending complete JSON object, that is:
   json_data = json.dumps(tweet)
   producer.send (topic, json_data)
   nothing prints on the console using pprint() function.

   Spark File has:

   os.environ[“SPARK_HOME”] = “C:spark”
   sc = SparkContext(appName=”PythonStreamingDirectKafkaWordCount”)
   ssc = StreamingContext(sc, 5)
   sc.setLogLevel(“ERROR”)
   brokers, topic = sys.argv[1:]
   print (“creating Kafka Direct Stream Object”)
   kvs = KafkaUtils.createDirectStream(ssc, [topic], {“metadata.broker.list”: brokers})
   print (“Json object extracted from kvs map”)
   tweetRDD = kvs.map(lambda (k, v): json.loads(v)).map(lambda tweet: tweet[“text”])
   print(“TweetRDD Created”)
   tweetRDD.pprint()
   print(“Should Print TweetRDD”)
   ssc.start()
   ssc.awaitTermination()

   Producer File:
   import tweepy
   from kafka import KafkaProducer
   import json

   class StdOutListener(tweepy.StreamListener):

   def on_status(self, tweet):
   try:
   print tweet.text
   json_data = json.dumps(tweet)
   producer.send(mytopic, json_data)
   ………
   if __name__ == ‘__main__’:
   producer = KafkaProducer(bootstrap_servers=”localhost:9092″, value_serializer=lambda v: json.dumps(v).encode(‘utf-8’))

   Reply