All Questions
4 questions
1
vote
1
answer
1k
views
Big files causing shuffle error in hadoop map reduce
I am seeing the following error when I try to process big file like size > 35GB files, but doesn't happen when I try less big file like size < 10GB .
App > Error: org.apache.hadoop.mapreduce....
- 2,292
2
votes
1
answer
137
views
Debug failed shuffles in hadoop map reduces
I am seeing as the size of the input file increase failed shuffles increases and job complete time increases non linearly.
eg.
75GB took 1h
86GB took 5h
I also see average shuffle time increase 10 ...
- 2,292
2
votes
2
answers
6k
views
Fixing java.lang.NoSuchMethodError: com.amazonaws.util.StringUtils.trim
Consider the following error:
2018-07-12 22:46:36,087 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: com.amazonaws.util.StringUtils.trim(Ljava/...
- 2,292
5
votes
1
answer
2k
views
How to kill hadoop job gracefully/intercept `hadoop job -kill`
My Java application runs on mapper and creates child processes using Qubole API. Application stores child qubole queryIDs. I need to intercept kill signal and shutdown child processes before exit. ...
- 38.2k