#NoPayJan Offer - Access all CloudxLab Courses for free between 1st to 31st Jan

  Enroll Now >>

Apache Spark - Counting Word Frequencies

Note - In this video, we used Hue to access the results in HDFS. We have deprecated the Hue. Please use the below commands in the web console to access the files

  • Login to the web console
  • Check the files

    hadoop fs -ls  my_result
    
  • Check the content of the first part

    hadoop fs -cat my_result/part-00000 | more
    
  • Check the content of the second part

    hadoop fs -cat my_result/part-00001 | more
    
  • Given below is the Scala code for counting word frequencies

    var linesRdd = sc.textFile("/data/mr/wordcount/input/big.txt")
    var words = linesRdd.flatMap(x => x.split(" "))
    var wordsKv = words.map(x => (x, 1))
    //def myfunc(x:Int, y:Int): Int = x + y
    var output = wordsKv.reduceByKey(_ + _)
    output.take(10)
    

    or

    output.saveAsTextFile("my_result")
    

No hints are availble for this assesment

Answer is not availble for this assesment

Loading comments...