Updated on 2022-09-23 GMT+08:00

Developing an MRS Flink Job

This section describes how to develop an MRS Flink job on DataArts Factory. Use an MRS Flink job to count the number of words.

Prerequisites

  • You have the permission to access OBS paths.
  • MRS has been enabled and an MRS cluster has been created.

Data Preparation

Procedure

  1. Upload the job resource package and data file to the OBS bucket.

    In this example, upload WordCount.jar to lkj_test/WordCount.jar and word.txt to lkj_test/input/word.txt.

  2. Create an empty job named job_MRS_Flink.

    Figure 1 Creating a job

  3. Go to the job development page, drag the MRS Flink node to the canvas, and click the node to configure its properties.

    Figure 2 Configuring properties for an MRS Flink node

    Parameter descriptions:

    --Flink job name
    wordcount
    --MRS cluster name
    Select an MRS cluster.
    --Program parameter
    -c    org.apache.flink.streaming.examples.wordcount.WordCount
    --Flink job resource package
    wordcount
    --Input data path
    obs://dlf-test/lkj_test/input/word.txt
    --Output data path
    obs://dlf-test/lkj_test/output.txt

    Specifically:

    obs://dlf-test/lkj_test/input/word.txt is the directory where the wordcount.jar parameters are passed. You can pass the words to count.

    obs://dlf-test/lkj_test/output.txt is the directory where the output parameter file is stored. (If the output.txt file already exists, an error is reported.)

  4. Click Test to execute the MRS Flink job.
  5. After the test is complete, click Submit.
  6. Choose Monitor Job in the navigation pane and view the job execution result.
  7. View the returned records in the OBS bucket. (Skip this step if the return function is not configured.)