Spark Programming - How much memory is consumed for loading an RDD?...

45 / 52

How much memory is consumed for loading an RDD?

Spark will consume whatever memory is needed by RDD partition. If memory is not available, it will fail.
Spark will dedicate 60% of the memory to RDD. If RDD is not fitting, older RDD partitions will be dropped and loaded again when needed.
Spark will only consume memory for a single record that is being processed while keeping rest of records on disk