Querying Logs for Flink Jobs

Scenario

DLI job buckets are used to store temporary data generated during DLI job running, such as job logs and results.

This section describes how to configure a bucket for DLI jobs on the DLI console and obtain Flink job logs.

Notes

• To avoid disordered job results, do not use the OBS bucket configured for DLI jobs for any other purposes.
• DLI jobs must be set and modified by the main account as IAM users do not have required permissions.
• You cannot view the logs for DLI jobs before configuring a bucket.
• You can configure lifecycle rules to periodically delete objects from buckets or change storage classes of objects.
• Exercise caution when modifying the job bucket, as it may result in the inability to retrieve historical data.

Prerequisites

Before the configuration, create an OBS bucket or parallel file system (PFS). In big data scenarios, you are advised to create a PFS. PFS is a high-performance file system provided by OBS, with access latency in milliseconds. PFS can achieve a bandwidth performance of up to TB/s and millions of IOPS, which makes it ideal for processing high-performance computing (HPC) workloads.

For details about PFS, see "Parallel File System Feature Guide" in the Object Storage Service User Guide.

Configuring a Bucket for DLI Jobs

1. In the navigation pane of the DLI console, choose Global Configuration > Project.
2. On the Project page, click next to Job Bucket to configure bucket information.
   Figure 1 Project
   
3. Click to view available buckets.
4. In the displayed OBS dialog box, click the name of a bucket or search for and click a bucket name and then click OK. In the Set Job Bucket dialog box, click OK.
   Temporary data generated during DLI job running will be stored in the OBS bucket.

   Figure 2 Setting the job bucket
   

Viewing Commit Logs

You can check commit logs to locate commit faults.

1. In the navigation pane of the DLI console, choose Job Management > Flink Jobs.
2. Click the name of the Flink job whose commit logs you want to check.
3. Click the Commit Logs tab and check the job commit process.
   Figure 3 Commit logs
   

Viewing Run Logs

You can check run logs to locate job running faults.

1. In the navigation pane of the DLI console, choose Job Management > Flink Jobs.
2. Click the name of the Flink job whose commit logs you want to check.
3. Click the Run Log tab and check the JobManager and TaskManager information of the running job.
   Figure 4 Run logs
   

   JobManager and TaskManager information is updated every minute. By default, the run logs generated in the last minute are displayed.

   If you have configured an OBS bucket to store job logs, you can access it to download and check historical logs.

   

   For how to upload files to an OBS bucket, see Uploading an Object in the Object Storage Service Getting Started.

   If the job is not running, you cannot check Task Manager information.