Spark Memory Overhead

Question

Spark memory overhead related question asked multiple times in SO, I went through most of them. However, after going through multiple blogs, I got confused.

Below are the questions I have

• whether memory overhead is part of the executor memory or it's separate? As few of the blogs are saying memory overhead is part of the executor memory and others are saying executor memory + memory overhead(is that mean memory overhead is not part of the executor memory)?
• Memory overhead and off-heap over are the same?
• What happens if I didn't mention overhead as part of the spark-submit, will it take default 18.75 or it won't?
• Will there be any side effects if we give more memory overhead than the default value?

https://docs.qubole.com/en/latest/user-guide/engines/spark/defaults-executors.html https://spoddutur.github.io/spark-notes/distribution_of_executors_cores_and_memory_for_spark_application.html

Below is the case I want to understand. I have 5 nodes with each node 16 vcores and 128GB Memory(out of which 120 is usable), now I want to submit spark application, below is the conf, I'm thinking

Total Cores 16 * 5 = 80
Total Memory 120 * 5 = 600GB

case 1: Memory Overhead part of the executor memory

spark.executor.memory=32G
spark.executor.cores=5
spark.executor.instances=14 (1 for AM)
spark.executor.memoryOverhead=8G ( giving more than 18.75% which is default)
spark.driver.memoryOverhead=8G
spark.driver.cores=5

Case 2: Memory Overhead not part of the executor memory

spark.executor.memory=28G
spark.executor.cores=5
spark.executor.instances=14 (1 for AM)
spark.executor.memoryOverhead=6G ( giving more than 18.75% which is default)
spark.driver.memoryOverhead=6G
spark.driver.cores=5

As per the below video, I'm trying to use 85% of the node i.e. around 100GB out of 120GB, not sure if we can use more than that.

https://www.youtube.com/watch?v=ph_2xwVjCGs&list=PLdqfPU6gm4b9bJEb7crUwdkpprPLseCOB&index=8&t=1281s (4:12)

Answer 1

To answer your question whether memory overhead is part of the executor memory or it's separate? Memory Overhead is not part of executor memory.

Resource manager launches containers in order to execute executors inside it. so basically executor memory + memory overhead = container memory ..... spark have break up for executor memory into application memory and cache memory.

Executor memory overhead mainly includes off-heap memory and nio buffers and memory for running container-specific threads(thread stacks). when you do not specify memory overhead, Resource manager calculates memory overhead value by using default values and launch containers accordingly.

It is always recommended to keep aside cores and memory for OS (which is 1 core for nodemanager and 1 core for other daemons and 2 cores for OS to work optimally)

You can change your calculation like below mentioned 12 * 5 = 60cores and total memory 116 * 5 = 580GB is what total resources available .. then you tune other parameters correspondingly.

Answer 2

1. Memory overhead is not part of the executor memory. Executor memory is set by spark.executor.memory. Memory overhead is part of the container memory.
2. Both memory overhead and the amount of memory defined by spark.memory.offHeap.size are allocated outside the JVM Heap. The below image, sourced from the medium article mentioned below, clearly depicts the definition and usage of both.
3. There will be a default value for the amount memory overhead used:

max(executorMemory * spark.executor.memoryOverheadFactor, 384 MiB)

4. As this portion is allocated from the off-heap memory, there will not be any garbage collection. So we might have to be care in managing the memory.

Reference: https://medium.com/walmartglobaltech/decoding-memory-in-spark-parameters-that-are-often-confused-c11be7488a24