Updated on 2024-08-12 GMT+08:00

Downloading and Importing Sample Projects

Scenario

Download sample projects and import them to the IDEA on Windows for learning.

Prerequisites

  • A developer account, for example, developuser, has been prepared by following instructions in Preparing Development and Operating Environment, and the user authentication credential file has been downloaded to the local host.

    The user has the common user permission of Oozie, HDFS access permission, Hive table read and write permission, HBase read and write permission, and Yarn queue submission permission.

  • A complete cluster client has been installed in the Linux environment.
  • The URL of a running Oozie server (any node) has been obtained. The URL is the target address to which the client submits a workflow job.

    The URL format is https://Oozie node service IP address:21003/oozie. The port number is the value of OOZIE_HTTPS_PORT, which is 21003 by default.

    for example, https://10.10.10.176:21003/oozie.

Procedure

  1. Obtain the OozieMapReduceExample, OozieSparkHBaseExample, and OozieSparkHiveExample sample projects from the sample project folder ooziesecurity-examples in the src\oozie-examples directory where the sample code is decompressed. For details, see Obtaining the MRS Application Development Sample Project.
  2. Copy the keytab file user.keytab and user authentication credential file krb5.conf obtained in Preparing a Developer Account to the \src\main\resources directory of the OozieMapReduceExample, OozieSparkHBaseExample, and OozieSparkHiveExample sample projects.
  3. In an application development environment, import the sample projects to the IDEA development environment.

    1. Choose File > Open.

      The Browse dialog box is displayed.

    2. Select a sample project folder, and click OK.

  4. Modify the parameters in the sample project. For details, see Table 1.

    Table 1 Parameters to be modified

    File Name

    Parameter

    Value

    Example Value

    \src\main\resources\job.properties

    userName

    User who submits a job.

    developuser

    \src\main\resources\application.properties

    submit_user

    User who submits a job.

    developuser

    oozie_url_default

    https://Oozie service IP address:21003/oozie

    https://10.10.10.176:21003/oozie

  5. Select the sample project to be run.

  6. Use the client to upload the examples folder of Oozie to HDFS.

    1. Log in to the node where the client is installed, and switch to the directory of the client, for example /opt/client.

      cd /opt/client

    2. Run the following command to configure environment variables:

      source bigdata_env

    3. Run the following command to perform user authentication. Change the password upon the first login.

      kinit developuser

    4. Run the following commands to create a directory in HDFS and upload the sample project to the directory:

      hdfs dfs -mkdir /user/developuser

      hdfs dfs -put -f /opt/client/Oozie/oozie-client-*/examples /user/developuser

      In the preceding command, oozie-client-* indicates the version number. Replace it with the actual version number.