Apache Spark - Lazy Evaluation & Lineage Graph

Not able to play video? Try with vimeo

INSTRUCTIONS

Actions: Lazy Evaluation - Optimization - Scala

Individual map transformations

def Map1(x:String):String = x.trim();
def Map2(x:String):String =  x.toUpperCase();
var lines = sc.textFile(...)
var lines1 = lines.map(Map1);
var lines2 = lines1.map(Map2);
lines2.collect()

...converted by Spark into single transformation because of lazy evaluation

def Map3(x:String):String={
    var y = x.trim();
    return y.toUpperCase();
}
lines = sc.textFile(...)
lines2 = lines.map(Map3);
lines2.collect()

Lineage graph

lines  = sc.textFile("myfile");
fewlines = lines.filter(...)
uppercaselines = fewlines.map(...)
uppercaselines = fewlines.map(...)
uppercaselines.count()