Downloading and Importing Sample Projects

Scenario

Download sample projects and import them to the IDEA on Windows for learning.

Prerequisites

• A developer account, for example, developuser, has been prepared by following instructions in Preparing Development and Operating Environment, and the user authentication credential file has been downloaded to the local host.

  The user has the common user permission of Oozie, HDFS access permission, Hive table read and write permission, HBase read and write permission, and Yarn queue submission permission.

• A complete cluster client has been installed in the Linux environment.
• The URL of a running Oozie server (any node) has been obtained. The URL is the target address to which the client submits a workflow job.

  The URL format is https://Oozie node service IP address:21003/oozie. The port number is the value of OOZIE_HTTPS_PORT, which is 21003 by default.

  for example, https://10.10.10.176:21003/oozie.

Procedure

1. Obtain the OozieMapReduceExample, OozieSparkHBaseExample, and OozieSparkHiveExample sample projects from the sample project folder ooziesecurity-examples in the src\oozie-examples directory where the sample code is decompressed. For details, see Obtaining the MRS Application Development Sample Project.
2. Copy the keytab file user.keytab and user authentication credential file krb5.conf obtained in Preparing MRS Application Development User to the \src\main\resources directory of the OozieMapReduceExample, OozieSparkHBaseExample, and OozieSparkHiveExample sample projects.
3. In an application development environment, import the sample projects to the IDEA development environment.
   1. Choose File > Open.

      The Browse dialog box is displayed.

   2. Select a sample project folder, and click OK.

4. Modify the parameters in the sample project. For details, see Table 1.

   Table 1 Parameters to be modified

   File Name	Parameter	Value	Example Value
   \src\main\resources\job.properties	userName	User who submits a job.	developuser
   \src\main\resources\application.properties	submit_user	User who submits a job.	developuser
   	oozie_url_default	https://Oozie service IP address:21003/oozie	https://10.10.10.176:21003/oozie

5. Select the sample project to be run.
   • For the OozieMapReduceExcample sample project, go to 6.
   • For the OozieSparkHBaseExample and OozieSparkHiveExample sample projects, see Scheduling Spark2x to Access HBase and Hive Using Oozie.

6. Use the client to upload the examples folder of Oozie to HDFS.
   1. Log in to the node where the client is installed, and switch to the directory of the client, for example /opt/client.

      cd /opt/client

   2. Run the following command to configure environment variables:

      source bigdata_env

   3. Run the following command to perform user authentication. Change the password upon the first login.

      kinit developuser

   4. Run the following commands to create a directory in HDFS and upload the sample project to the directory:

      hdfs dfs -mkdir /user/developuser

      hdfs dfs -put -f /opt/client/Oozie/oozie-client-*/examples /user/developuser

      

      In the preceding command, oozie-client-* indicates the version number. Replace it with the actual version number.