On this page

Show all

Help Center/ Data Lake Insight/ User Guide/ Submitting a Flink Job Using DLI/ Managing Flink Jobs/ Querying Logs for Flink Jobs

Querying Logs for Flink Jobs

Updated on 2025-03-04 GMT+08:00

View PDF

Scenario

DLI job buckets are used to store temporary data generated during DLI job running, such as job logs and results.

This section describes how to configure a bucket for DLI jobs on the DLI console and obtain Flink job logs.

Notes

To avoid disordered job results, do not use the OBS bucket configured for DLI jobs for any other purposes.
DLI jobs must be set and modified by the main account as IAM users do not have required permissions.
You cannot view the logs for DLI jobs before configuring a bucket.
You can configure lifecycle rules to periodically delete objects from buckets or change storage classes of objects.
Exercise caution when modifying the job bucket, as it may result in the inability to retrieve historical data.

Prerequisites

Before the configuration, create an OBS bucket or parallel file system (PFS). In big data scenarios, you are advised to create a PFS. PFS is a high-performance file system provided by OBS, with access latency in milliseconds. PFS can achieve a bandwidth performance of up to TB/s and millions of IOPS, which makes it ideal for processing high-performance computing (HPC) workloads.

For details about PFS, see "Parallel File System Feature Guide" in the Object Storage Service User Guide.

Configuring a Bucket for DLI Jobs

In the navigation pane of the DLI console, choose Global Configuration > Project.
On the Project page, click next to Job Bucket to configure bucket information.
Figure 1 Project
Click to view available buckets.
In the displayed OBS dialog box, click the name of a bucket or search for and click a bucket name and then click OK. In the Set Job Bucket dialog box, click OK.
Temporary data generated during DLI job running will be stored in the OBS bucket.
Figure 2 Setting the job bucket

Viewing Commit Logs

You can check commit logs to locate commit faults.

In the navigation pane of the DLI console, choose Job Management > Flink Jobs.
Click the name of the Flink job whose commit logs you want to check.
Click the Commit Logs tab and check the job commit process.

Figure 3 Commit logs

Viewing Run Logs

You can check run logs to locate job running faults.

In the navigation pane of the DLI console, choose Job Management > Flink Jobs.
Click the name of the Flink job whose commit logs you want to check.
Click the Run Log tab and check the JobManager and TaskManager information of the running job.

Figure 4 Run logs

JobManager and TaskManager information is updated every minute. By default, the run logs generated in the last minute are displayed.

If you have configured an OBS bucket to store job logs, you can access it to download and check historical logs.

NOTE:

For how to upload files to an OBS bucket, see Uploading an Object in the Object Storage Service Getting Started.

If the job is not running, you cannot check Task Manager information.

Viewing the Log List

You can view historical job files by viewing the Flink job log list.

In the navigation pane of the DLI console, choose Job Management > Flink Jobs.
Click the name of your desired job.
On the Logs tab page, choose Job Manager and Task Manager separately to view their logs.

Figure 5 Flink job log list

Parent topic: Managing Flink Jobs

Previous topic: Enabling Dynamic Scaling for Flink Jobs

Next topic: Common Operations of Flink Jobs

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot