Help Center/ MapReduce Service/ User Guide (Paris Region)/ Configuring a Cluster with Storage and Compute Decoupled/ Using a Storage-Compute Decoupled Cluster/ Interconnecting Flink with OBS
Updated on 2024-10-11 GMT+08:00
View PDF
Share

Interconnecting Flink with OBS

Before performing the following operations, ensure that you have configured a storage-compute decoupled cluster by referring to Configuring a Storage-Compute Decoupled Cluster (Agency) or Configuring a Storage-Compute Decoupled Cluster (AK/SK).

  1. Log in to the Flink client installation node as the client installation user.
  2. Run the following command to initialize environment variables:

    source ${client_home}/bigdata_env

  3. Configure the Flink client properly. For details, see Installing a Client (Version 3.x or Later).
  4. For a security cluster, run the following command to perform user authentication. If Kerberos authentication is not enabled for the current cluster, you do not need to run this command.

    kinit Username

  5. Explicitly add the OBS file system to be accessed in the Flink command line.

    ./bin/flink run --class com.xxx.bigdata.flink.examples.FlinkProcessingTimeAPIMain ./config/FlinkCheckpointJavaExample.jar --chkPath obs://Name of the OBS parallel file system

Flink jobs are running on Yarn. Before configuring Flink to interconnect with the OBS file system, ensure that the interconnection between Yarn and the OBS file system is normal.