Spark Programming - Spark reserves 20% of memory to shuffle and agregation buffers.... | Automated hands-on| CloudxLab

Advanced Spark Programming with Python

43 / 49

Previous Index Next

Spark reserves 20% of memory to shuffle and agregation buffers. What are these buffer?

These are the temporary data structures required while we invoking operations such as reduceByKey, sort etc.
The libraries that your program depends-on is loaded into this buffer

Note - Having trouble with the assessment engine? Follow the steps listed here

Previous Index Next

Loading comments...