Help Center/ MapReduce Service/ Best Practices/ Data Analytics/ Using Flink Jobs to Process OBS Data
Updated on 2026-09-07 GMT+08:00

Using Flink Jobs to Process OBS Data

Application Scenarios

In the big data processing scenario of separating storage and computing, users store massive amounts of data in OBS (Object Storage Service) and utilize the elastic computing power of the MRS cluster to process the data. This approach can effectively reduce storage costs while meeting the demand for on-demand scaling of computing resources. So, how can Flink jobs efficiently process the stored data in OBS?

MRS supports decoupled storage and compute in scenarios where a large storage capacity is required and compute resources need to be scaled on demand. This allows you to store your data in OBS and use an MRS cluster only for data computing.

This practice instructs you on how to run Flink jobs in an MRS cluster to process data stored in OBS.

Solution Architecture

Flink is a unified computing framework that supports both batch processing and stream processing. It provides a stream data processing engine that supports data distribution and parallel computing. Flink features stream processing and is a top open-source stream processing engine in the industry.

Flink provides high-concurrency pipeline data processing, millisecond-level latency, and high reliability, making it suitable for low-latency data processing.

Figure 1 Flink Architecture

In this example, the Flink WordCount job program built in the MRS cluster is used to analyze the source data stored in the OBS file system and compute the frequency of words in the source data.

You can also obtain MRS sample code project and develop other Flink stream job programs by referring to Flink Development Guide.

Procedure

The operation process is as follows:

  1. Step 1: Creating an MRS Cluster: Create an MRS cluster that contains the Flink component.
  2. Step 2: Preparing Test Data: Prepare the source data and program for analysis and upload them to OBS.
  3. Step 3: Creating and Running a Flink Job: Create and run a Flink job in the MRS cluster.
  4. Step 4: Viewing Job Execution Results: View the job execution result.
Figure 2 Operation Process

Step 1: Creating an MRS Cluster

Create and purchase an MRS cluster that contains the Flink component to provide a computing environment for running Flink jobs. For details, see Buying a Custom Cluster.

In this practice, an MRS 3.6.0-LTS.1 cluster with Kerberos authentication disabled is used as an example.

In this example, before you analyze data stored in OBS, bind an IAM agency to the MRS cluster so that cluster components can connect to the OBS file system and have operation permissions on file system directories.

You can select the default MRS_ECS_DEFAULT_AGENCY agency or create a custom agency that has the permission to operate the OBS file system.

Figure 3 Setting up an MRS Cluster Agency

After the cluster is purchased, install the cluster client on any node of the cluster as user root. For details, see Installing and Using the Cluster Client.

Assume that the client is installed in /opt/client.

Step 2: Preparing Test Data

Before you create a Flink job for data analysis, prepare test data to be analyzed and upload the data to OBS.

  1. Create a file named mrs_flink_test.txt on your local PC. For example, the file content is as follows:

    This is a test demo for MRS Flink. Flink is a unified computing framework that supports both batch processing and stream processing. It provides a stream data processing engine that supports data distribution and parallel computing. 

  2. Choose Service List > Storage > Object Storage Service.
  3. Create a parallel file system and upload test data files to provide data sources for Flink jobs. On the OBS management console that is displayed, choose Parallel File Systems in the navigation pane on the left. On the page displayed, click Create Parallel File System and set required parameters to create a parallel file system. After the system is created, upload the test data to it.

    For example, if the created file system is named mrs-demo-data, click the system name, and click the Files tab. On this tab page, click Create Folder to create a folder named flink and upload the test data to the folder.

    In this example, the complete path of the test data is obs://mrs-demo-data/flink/mrs_flink_test.txt.

  4. (Optional) Uploading Data Analysis Applications

    You can upload the JAR files of the Flink applications developed by yourself to OBS or HDFS of the MRS cluster.

    In this example, the Flink WordCount sample program built in the MRS cluster is used. You can obtain the sample program from the MRS cluster client installation directory, that is, /opt/client/Flink/flink/examples/batch/WordCount.jar.

    Upload WordCount.jar to the obs://mrs-demo-data/program directory.

Step 3: Creating and Running a Flink Job

Method 1: Submit a job online on the console.

  1. Log in to the MRS console.
  2. Click the cluster name to go to the cluster details page.
  3. On the Dashboard tab page, click Synchronize next to IAM User Sync to synchronize IAM users.
  4. Click the Jobs tab.
  5. Click Create. In the Create Job dialog box that is displayed, set the following parameters to create a Flink job.

    • Type: Select Flink.
    • Name: Customize a job name, for example, flink_obs_test.
    • Program Path: In this example, the WordCount program of the Flink client is used.
    • Program Parameter: Use the default value.
    • Parameters: Enter the input and output parameters of the application. The input parameter indicates the test data to be analyzed, and the output parameter indicates the result output file.

      In this example, set this parameter to --input obs://mrs-demo-data/flink/mrs_flink_test.txt --output obs://mrs-demo-data/flink/output.

    • Service Parameter: Use the default values. For details about how to manually configure job parameters, see Running a Flink Job.
    Figure 4 Adding a Flink job

  6. Confirm the job configuration information and click OK.

    Figure 5 Viewing the job status

Method 2: Submit a job using the cluster client.

  1. Log in to the node where the cluster client is installed as user root and go to the client installation directory.

    cd /opt/client
    source bigdata_env

  2. Run the following command to check whether the cluster can access OBS:

    hdfs dfs -ls obs://mrs-demo-data/flink/

  3. Submit a Flink job and specify the source file data for consumption.

    flink run -m yarn-cluster /opt/client/Flink/flink/examples/batch/WordCount.jar --input obs://mrs-demo-data/flink/mrs_flink_test.txt --output obs://mrs-demo-data/flink/output2

    The execution result is as follows:

    ...
    Cluster started: Yarn cluster with application id application_1654672374562_0011
    Job has been submitted with JobID a89b561de5d0298cb2ba01fbc30338bc
    Program execution finished
    Job with JobID a89b561de5d0298cb2ba01fbc30338bc has finished.
    Job Runtime: 1200 ms

Step 4: Viewing Job Execution Results

  1. After the job is submitted, log in to Manager of the MRS cluster and choose Cluster > Services > Yarn.

    For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

  2. Click the link next to ResourceManager WebUI to access the native Yarn web UI. On the All Applications page that is displayed, choose Applications on the left, and view the job running status and run logs.

    Figure 6 Yarn WebUI

  3. After the job execution is complete, you can view the data analysis result in the specified result output file in the OBS file system.

    Figure 7 Viewing the output result

    Check whether the operation is successful:

    Download the output file to the local PC and open it using a text editor. The file contains the word count result, indicating that the Flink job is successfully executed.

    a 3
    and 2
    batch 1
    both 1
    computing 2
    data 2
    demo 1
    distribution 1
    engine 1
    flink 2
    for 1
    framework 1
    is 2
    it 1
    mrs 1
    parallel 1
    processing 3
    provides 1
    stream 2
    supports 2
    test 1
    that 2
    this 1
    unified 1

    If you do not specify the output directory when submitting a job using the cluster client CLI, you can view the data analysis result on the job running page.

    Job with JobID xxx has finished.
    Job Runtime: xxx ms
    Accumulator Results: 
    - e6209f96ffa423974f8c7043821814e9 (java.util.ArrayList) [31 elements]
    
    (a,3)
    (and,2)
    (batch,1)
    (both,1)
    (computing,2)
    (data,2)
    (demo,1)
    (distribution,1)
    (engine,1)
    (flink,2)
    (for,1)
    (framework,1)
    (is,2)
    (it,1)
    (mrs,1)
    (parallel,1)
    (processing,3)
    (provides,1)
    (stream,2)
    (supports,2)
    (test,1)
    (that,2)
    (this,1)
    (unified,1)

FAQs

  • What should I do if a Flink job fails to be submitted?

    Check whether the MRS cluster has been correctly bound to an IAM agency, whether the OBS file system path is correct, whether the Flink job parameters are correctly configured, and whether the cluster resources are sufficient.

  • What should I do if a Flink job cannot read data from OBS?

    Check whether the MRS cluster has been bound to an IAM agency with the OBS operation permissions, whether the OBS file system path is correct, whether the test data file has been uploaded to the specified path, and whether the cluster node can access OBS.

Helpful Links