[Oozie - Introduction]
Oozie is a Java Web application used to schedule Apache Hadoop jobs. Oozie combines multiple jobs sequentially into one logical unit of work and executes them. It Is integrated with the rest of the Hadoop stack. It can execute Hadoop jobs out of the box such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop and Distcp as well as system specific jobs such as Java programs and shell scripts.
[Oozie - Jobs]
There are two types of Oozie jobs:
[Oozie - Jobs - Workflow Jobs]
Oozie Workflow jobs are Directed Acyclical Graphs - DAGs, specifying a sequence of actions to execute. DAG is a finite directed graph with no cycles. As shown in the image task 10 can only be executed after task 11 and 3 are executed.
Examples - Almost all task execution systems use DAG. Most Source Control Management Systems implement the revisions as a DAG
[Oozie - Jobs - Coordinator Jobs]
Oozie Coordinator jobs are recurrent Oozie Workflow jobs that are triggered by time and data availability.
Example - Let’s say if we want to take data from HDFS and put to Hive every one hour, we can define a workflow in Oozie to take data from HDFS and put to Hive and run it as a coordinator job.
[Oozie - Use Case]
Let's understand a use case where we will be using Oozie.
Let's say we want to push web server access logs to HDFS and run Spark MLlib recommendation algorithm every day to generate recommendations and display it to the user. To do this, typical steps will be
In this use case, Steps 2-7 can be run as Oozie Coordinator jobs daily.
[Oozie - Workflow - XML]
We define Oozie workflow in XML files. Let’s see a sample XML for MapReduce operation. We define map class, reduce class, input and output directories in XML. We can configure similar workflows for other actions like Hive, Pig etc