0

I do not understand this option. It seems like its the maximum number of executors.

If there is not enough memory on the nodes in the cluster this number is doing nothing and there are fewer executors than I'm asking for.

Can someone clarify this please?

Improve this question

1 Answer 1

Reset to default
0

The option --num-executors is used after we calculate the number of executors our infrastructure supports from the available memory on the worker nodes. The calculation can be performed as stated here. This wuill let you know the number of executors supported by your hadoop infrastructure or your the queue that has been assigned to your team.

Improve this answer
3
  • That is not what I'm asking. I want to know if its a maximum or minimum of executors number
    – idan ahal
    Commented May 8, 2022 at 20:52
  • 1
    It is the maximum number of executors that you want for your job to run on the cluster. The above calculations let you get a number of executors that your cluster will accept
    – Yayati Sule
    Commented May 9, 2022 at 2:41
  • But why will I give a maximum number? I use the 'Dominante resource calculator' and dynamic allocation, I defined executor-cores and executor-memory. Isn't it (In this case) useless? I'm asking if --num-executors is useless in this case of course.
    – idan ahal
    Commented May 9, 2022 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.