Interconnecting Flink with OBS Using an IAM Agency

After a cluster with decoupled storage and compute is configured by referring to Interconnecting an MRS Cluster with OBS Using an IAM Agency, you can access the OBS parallel file system and run jobs in a Flink job. This section describes how to submit a Flink WordCount job.

Prerequisites

Before interconnecting Flink with OBS, ensure that YARN is connected to OBS as Flink jobs run on YARN.

Interconnecting Flink with OBS

Log in to the Flink client installation node as the client installation user.

For details about how to download and install the cluster client, see Installing an MRS Cluster Client.

Initialize environment variables.

source Client installation directory/bigdata_env

Configure the Flink client. For details, see Using the Flink Client.
Start a session.
- Normal cluster (Kerberos authentication disabled)
```
yarn-session.sh -nm "session-name" -d
```
- Security cluster (Kerberos authentication enabled)
  - If the paths of the flink.keystore and flink.truststore files are relative ones:
    Run the following command in the directory at the same level as ssl to start the session. ssl/ is a relative path.
    
    Go to the conf directory.
```
cd Client installation directory/Flink/flink/conf
```
    Start a session.
```
yarn-session.sh -t ssl/ -nm "session-name" -d
```
    The session is started successfully.
```
...
Cluster started: Yarn cluster with application id application_1624937999496_0017
JobManager Web Interface: http://192.168.1.150:32261
```
  - If the paths of the flink.keystore and flink.truststore files are absolute paths, run the following commands to start a session:
    Go to the conf directory.
```
cd Client installation directory/Flink/flink/conf
```
    Start a session.
```
yarn-session.sh -nm "session-name" -d
```
Authenticate the user of the cluster with Kerberos authentication enabled. Skip this step for the user of the cluster with Kerberos authentication disabled.
```
kinit Username
```

Add the target OBS file system to the Flink CLI and execute the analysis program.

Create the test file in the /tmp/test directory.
```
echo -e 'test' >/tmp/test
```
Create a directory for storing the test file in the OBS parallel file system.
```
hdfs dfs -mkdir -p obs://OBS parallel file system name/tmp/flinkjob
```

Upload the test file to the OBS parallel file system.

hdfs dfs -put /tmp/test/ obs://OBS parallel file system name/tmp/flinkjob/

Run WordCount.jar.

flink run Client installation directory/Flink/flink/examples/batch/WordCount.jar -input obs://OBS parallel file system name/tmp/flinkjob/test -output obs://OBS parallel file system name/tmp/flinkjob/output