All Questions
1,100 questions
0
votes
1
answer
42
views
Java action in Apache Oozie workflow
I am trying to configure an Apache Oozie workflow to execute different actions depending on the day of the week. After reading https://stackoverflow.com/questions/71422257/oozie-coordinator-get-day-of-...
0
votes
0
answers
38
views
Apache oozie JA008 error - job state changed from SUCCEDED to FAILED
I'm running oozie HA 5.2.1 on EMR and I have an issue with this temporary directory. I have a workflow which has start node -> action node -> end node. The job start running -> runs for 10-15 ...
- 65
0
votes
1
answer
20
views
Apache Crunch Job On AWS EMR using Oozie
Context:
I want to run an apache crunch job on AWS EMR
this job is part of a pipeline of oozie java actions and oozie subworkflows (this particular job is part of a subworkflow). In oozie we have a ...
- 65
0
votes
0
answers
26
views
Incorrect behavior Oozie with increased load. Hanging subsidiaries in the status "RUNNING"
I launch an Oozie workflow with the following structure:
-- Oozie workflow
------> subworkflow_1
---------- fork_1
---------- fork_2
---------- ...
---------- fork_n
------> subworkflow_2
-------...
- 145
0
votes
0
answers
38
views
Oozie 5.2.1 with hadoop 2.7.2
Getting the following error when I try to validate my workflow.xml using oozie validate Workflow.xml
Error: E0701: XML schema error, /root/mr-wordcount/Workflow.xml, org.xml.sax....
0
votes
0
answers
49
views
what is the number of application masters spawned in oozie workflow
I have a Oozie workflow on Yarn comprising a spark action, optional shell action to collect logs in case of error and email notification action. When I try to run the workflow the spark action ends up ...
- 106
0
votes
1
answer
209
views
Not able to run(schedule) oozie example map reduce job || java.net.ConnectException
I am using Hadoop 2.6.0 and oozie 5.2.0 version.
Trying to run example Map reduce job in oozie but getting below error.
hadoop1@ip-172-31-84-37:/usr/local/oozie-5.2.0/examples/target/examples/apps/map-...
- 1
0
votes
0
answers
59
views
Sqoop command not found when running through Oozie
When I am running Sqoop script in CLI, it is running fine without any issue. But when run it using Oozie, it failed with Sqoop command not found. It seems sqoop is not installed in other data nodes. ...
- 185
0
votes
1
answer
33
views
Executing multiple mapred jobs that are executed in a sequence with a delay
I wanted to know if it was possible to execute/or submit a mapred job to be executed in the hadoop cluster after a given delay from the job before. To explain clearly, jobs are submitted in a sequence ...
- 3
0
votes
1
answer
253
views
Packaging shell action files with Oozie, retaining original directory structure
I have a PySpark application I would like to schedule with Oozie, using the shell action.
My submit-application.sh script simply initializes a Python virtualenv (present on all worker nodes) and calls ...
- 545
0
votes
1
answer
404
views
Running Python Script in OOzie with special libraries without install them to server
I want to run Python script in OOzie workflow with special libraries. But, I want to run this script without installing these special libraries to Hadoop nodes. I tried to run with virtualenv but ...
- 250
0
votes
0
answers
59
views
Oozie workflow fails to execute hive query using UDF with com.google.i18n.phonenumbers.PhoneNumberUtil.parse()
I have created a .jar file that utilizes many functions from google.i18n.phonenumbers java package. The jar file correctly loads the necessary external libraries from com.googlecode.libphonenumber....
- 1,539
-1
votes
1
answer
282
views
Is there a way to install apache oozie in windows 10
How I can install apache oozie in windows 10, not vm
I could not find any documents.
Thanks
1
vote
1
answer
381
views
Sqoop Import command throwing error but it is working from terminal
Below command is working fine from terminal but when I extecute it from Oozie it is throwing error.
bash
sqoop import --connect jdbc:mysql://172.16.16.128:3306/employees -- username=hive --password=...
- 84
1
vote
0
answers
182
views
Oozie workflow example failing despite running oozie-setup.sh sharelib with EJ001
I'm trying to run the examples documented at https://oozie.apache.org/docs/5.2.1/DG_Examples.html
When I run the first example it fails with EJ001:
root@50e1f53d5ded:/opt/oozie# oozie job -oozie http:/...
- 11
0
votes
1
answer
93
views
can OOZIE Map-Reduce job save data in parquet format?
I have a Map Reduce job that uses OOZIE workflow xml and writes output in sequence file format(org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat), is there something like this for saving ...
- 363
0
votes
0
answers
747
views
What is the Oozie alternative component in Google Cloud Platform (GCP)
I want to deploy the Hadoop based project in the Google Cloud Platform (GCP). At present, my project is being used multiple Big data stack like HDFS, Hive, Impala, Phoenix, HBase, Spark, Oozie etc. I ...
- 59
0
votes
0
answers
137
views
How to avoid prompting password with API AuthOozieClient - AuthType.KERBEROS
I have code as below to get the jobs status of current running Co-ordinators using OozieClient API from the different box which is not part of the cluster. Using AuthType as KERBEROS. When ever I am ...
- 74
1
vote
0
answers
228
views
Running Sqoop with Oozie Error: Can not create a Path from an empty string
I am trying to Run Sqoop export with Oozie. I can run simple Sqoop commands (list-tables etc) and I can run my Sqoop export command from the cmd line, however when I run with Oozie I get the following ...
- 11
0
votes
1
answer
224
views
Facing issue in setting up oozie with secure MapR cluster
We are facing an issue with setting up an oozie service with secure mapr cluster.
We are using the MapR installer to setup the MapR Cluster. Below are the configuration and steps that we followed.
...
- 31
0
votes
0
answers
827
views
The oozie job does not run with the message [AM container is launched, waiting for AM container to Register with RM]
I ran a shell job among the oozie examples.
However, YARN application is not executed.
Detail information YARN UI & LOG:
https://docs.google.com/document/d/1N8LBXZGttY3rhRTwv8cUEfK3WkWtvWJ-...
- 1
0
votes
0
answers
403
views
Schedule Python scripts with dependencies using Oozie
I'm working on a Cloudera environment using Oozie to schedule jobs.
Given that Oozie lets users schedule Shell scripts in Workflows, I have created the following .sh file in my Hadoop directory:
#! /...
- 607
0
votes
0
answers
185
views
Oozie workflow error E0501: Could not perform authorization operation
I'm using oozie 4.3.0 with hadoop 2.7.3.
When i want to create and run a workflow, i'm getting the following error:
Error: E0501 : E0501: Could not perform authorization operation, User: st_jsgane....
- 1
0
votes
0
answers
252
views
OOZIE Error: Could not find or load main class org.apache.oozie.tools.OozieDBCLI
I'm installing OOZIE 4.30 with Hadoop 2.9.1 and hive 1.2.2.
But when i do
$ ooziedb.sh create -sqlfile oozie.sql -run
i get
OOZIE Error: Could not find or load main class org.apache.oozie.tools....
- 1
0
votes
2
answers
90
views
oozie workflow throws Socket error but submits the workflow twice after 10 minutes
I am facing very weird issue. I have workflow xml which contains like 20 fork-join nodes and each contain 4-8 actions . When I submits this workflow, It wait for like 5-6 minutes, throws
"Error: ...
- 41
0
votes
1
answer
735
views
How to mark an Oozie workflow action's status as OK
I am using Apache oozie. I want to mark the status of one of the shell action as OK, in my oozie workflow. It is in Running state.
Can we please share the command to use in Apache Oozie to do this.
- 91
0
votes
1
answer
1k
views
Oozie java action fails with Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.exc.InputCoercionException
I am facing the below issue when I try to run oozie java action. This issue is not occurring in all clusters. On some specific clusters I see this issue.
Sharelib of oozie have lower version of ...
- 131
-2
votes
3
answers
2k
views
Hadoop 3.3 and oozie 5.2.0
I am using hadoop 3.3 and oozie 5.2.0.
I am getting below error:
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl....
1
vote
2
answers
3k
views
Get spark application id based on oozie job id
I am trying to get spark application id from unix based on oozie id. I am able to get map reduce job id when i try with oozie -info <oozie_id>@<action_name>. How can I get spark ...
- 363
1
vote
1
answer
393
views
Hadoop table source details
I have a hadoop table and I need to find the source type(relational/flatfile/mainframe) of the table. I have access to hue, logs of table. May I know how can I extract those details?
- 31
0
votes
1
answer
166
views
How to point centralized location for multiple workflows in oozie
I have more than 10 oozie workflows. Each workflow.xml, coordinator.properties and xml plus lib folder is in a separate folder. All the workflow have some common jars around 6mb size and I have to ...
- 704
0
votes
1
answer
421
views
How to get the first day of the month in input-events in oozie coordinator?
I need to configure data waiting for the whole previous month. How to get the first day of last month in start-instance if the start is on 2nd of month?
<input-events>
<data-in ...
- 41
1
vote
1
answer
899
views
Oozie spark-submit, `--driver-cores` parameter not working
I am doing spark submit from oozie, --driver-cores option is not working. For examples if i provided --driver-cores 4, yarn still creates 1 vCore container for driver.
Spark Opts in oozie:
<...
- 1,206
0
votes
1
answer
279
views
issues with Oozie and Sqoop Export
I am trying to do an Sqoop export, the sqoop command works just fine in the local Servers, however, when I try to use the same command as an Oozie action, I am getting the following error, any help ...
-1
votes
1
answer
165
views
Hadoop Oozie CPU utilization high
I'm currently running 20 concurrent jobs in OOZIE and noticed that oozie is hitting 400% of CPU utilization in the VM. How to reduce the load? Will adding another oozie server reduce the load on it?
- 1
3
votes
0
answers
992
views
Oozie spark action with kerberos
Unable to schedule oozie spark action with Kerberos cluster.
<workflow-app name="Spark Test" xmlns="uri:oozie:workflow:0.5">
<credentials>
<credential name="hiveCredentials" type=...
- 131
0
votes
1
answer
612
views
oozie shell script to execute beeline in kerberos cluster
I have been trying to execute a shell script that consists of beeline to execute hive queries.
The cluster is a Kerberos enabled cluster.
I am able to execute if script has kinit with keytab and ...
- 131
0
votes
1
answer
1k
views
Application master is killed by yarn while running spark job in cluster mode randomly
The error log is as follows :
20/05/10 18:40:47 ERROR yarn.Client: Application diagnostics message: Application application_1588683044535_1067 failed 2 times due to AM Container for ...
- 109
0
votes
0
answers
603
views
Oozie credentials kerberos
Oozie credentials for Kerberos cluster are working well with hive and hbase
Suppose consider an example where I have an oozie shell action that reads hdfs files. Oozie credentials are not coming to ...
- 131
0
votes
0
answers
270
views
java action to access kerberos Hive(SSL enabled)
I have been trying to access Hive(Kerberos and SSL enabled) server from java action.
I need to perform some Hive actions like accessing Hive database and tables, working with hive partitions etc.
...
- 131
0
votes
0
answers
134
views
Setting Kerberos Authentification to Oozie Coordinator
I'm doing a Spark Copy from a remote HDFS to my HDFS.
I have a Oozie Coordinator which check every day, if the data is available in the specified directory in the remote HDFS and then run the ...
- 165
1
vote
1
answer
958
views
Oozie - run a workflow every day or every hour
I have a oozie workflow(hive_insertion.xml) that executes a .hive file, which inserts data into a table.
The Oozie workflow is:
<workflow-app xmlns = "uri:oozie:workflow:0.4" name = "simple-...
- 3,093
1
vote
1
answer
3k
views
When Spark call Hive from oozie, exception raised "java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.metadata.HiveException"
I have spark job that save data to hdfs then it save the same data to Hive table. When I run it on Jupyter, it run succesfully. But when I run it through oozie It raises the folloing exception when it ...
- 31
0
votes
1
answer
902
views
How to copy file from local to HDFS directory in Oozie spark scala job?
I am trying to copy some files from local path to hdfs with scala, and running it with oozie. The job is failing as it is not able to read files from local path. Is there a way to read local files in ...
0
votes
1
answer
26
views
hadoop and its technologies setup
For study project requirement, I am selecting following technology because source of data is SQL SERVER
Initial data size is 100Gb and 10 growth@quarter
Information
Hadoop – Multi node cluster (...
0
votes
1
answer
288
views
Oozie loading jars provided by user first
I have tried oozie.launcher.mapreduce.job.user.classpath.first property in workflow.xml to load my jars first over sharelib jars.
Unable to run workflow when this property is used.
hadoop-hdfs is ...
- 131
0
votes
0
answers
321
views
Oozie Installation error-bin/mkdistro.sh -DskipTests
hduser@kas-XX:/usr/lib/Oozie/oozie-4.3.0$ bin/mkdistro.sh -DskipTests
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal....
0
votes
0
answers
108
views
oozie workflow to grant 777 permission on hive table ie directories
I wanted to grant 777 permission to hive directories using OOzie workflow.
do we have any xml tags which will be used directly
FYI I am using Hadoop , Ambari , HUE.
0
votes
0
answers
1k
views
oozie - java.lang.NoClassDefFoundError: Could not initialize class java.net.NetworkInterface
While running the oozie sample examples, Oozie jobs are getting scheduled and status showing as Running. After some times jobs get KILLED. While digging the hadoop logs found these exceptions.
I have ...
