Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Flink/ Open-Source Flink Features/ Configuring the Flink HistoryServer
Updated on 2026-06-27 GMT+08:00
View PDF
Share

Configuring the Flink HistoryServer

This section applies to MRS 3.6.0-LTS and later.

Introduction

Flink analyzes job failures by examining logs, which reveal causes, failed operators, and checkpoint status. To understand what led up to a job failure, use the HistoryServer to review pre-failure status, including specific operator failures, network conditions, execution plans, and checkpoint completions.

For completed jobs, the HistoryServer provides the data traffic, which facilitates issue location and Flink job maintenance.

Constraints on Flink HistoryServer

  • The HistoryServer displays only the status of jobs and does not display specific logs.
  • The HistoryServer displays only the jobs that have been archived and configured with an archive path (for details, see Starting the Flink HistoryServer), including the following:
    • Failed jobs.
    • Jobs stopped by the Flink stop or cancel command.
  • The HistoryServer does not display the unarchived jobs, including the ones listed below:
    • Jobs that are terminated by Yarn applications.
    • Jobs stopped or deleted on the FlinkServer web UI.
    • Running jobs.

Starting the Flink HistoryServer

  1. Install the cluster client, for example, in /opt/hadoopclient.
  2. Log in to the node where the client is installed as the client installation user and run the following command to switch to the client installation directory:

    cd /opt/hadoopclient

  3. Run the following command to configure environment variables:

    source bigdata_env

  4. If Kerberos authentication has been enabled for the cluster (in security mode), run the following command for user authentication: If Kerberos authentication is disabled for the current cluster (in normal mode), skip this step.

    kinit Component service user

    Example: kinit flinkuser

  5. Run the following command to initialize environment variables:

    source /opt/hadoopclient/bigdata_env

  6. Manually create a directory on HDFS for job archiving and grant the required permission. For example, set the owner to flink, owner group to hadoop, and permission to 770.

    hdfs dfs -mkdir -p hdfs://hacluster/flink/history

    hdfs dfs -chmod 770 hdfs://hacluster/flink/history

    hdfs dfs -chown flink:hadoop hdfs://hacluster/flink/history

  7. Add the following parameters to the Client installation directory/Flink/flink/conf/flink-conf.yaml file and save it.

    Table 1 Mandatory HistoryServer parameters

    Parameter

    Description

    Example

    Suggestion

    historyserver.archive.fs.dir

    Perform 6 to create directories for archiving jobs. Separate the directories with commas (,). HistoryServer monitors archived jobs in these directories.

    hdfs://hacluster/flink/history

    -

    historyserver.web.address

    IP address of the HistoryServer web UI.

    192.168.x.x

    Use the IP address of the current node.

    historyserver.web.port

    HistoryServer web UI port

    32651

    32651-32661

    historyserver.web.tmpdir

    Local directory used by the HistoryServer REST API for storing temporary files.

    /tmp

    /opt/Bigdata/tmp

    To prevent the /tmp directory from filling up, use the /opt/Bigdata/tmp directory.
    Table 2 Mandatory job parameters

    Parameter

    Description

    Example

    jobmanager.archive.fs.dir

    file system directory where the completed job information will be uploaded to.

    Only jobs with this directory configured will be archived. Jobs without this configuration cannot be viewed on the HistoryServer.

    The value must match what is set for historyserver.archive.fs.dir.

    The HistoryServer retrieves the job status based on the archived files in this directory.

    hdfs://hacluster/flink/history
    Table 3 Optional HistoryServer parameters

    Parameter

    Description

    Default Value

    Recommended Value

    historyserver.archive.clean-expired-jobs

    Whether to clean up jobs that are no longer archived to historyserver.archive.fs.dir

    false

    false

    historyserver.archive.fs.refresh-interval

    Interval for refreshing the archive directory, in milliseconds.

    10000

    10000

    historyserver.archive.retained-jobs

    The maximum number of jobs to be retained in each archive directory specified in historyserver.archive.fs.dir. Value -1 (default) indicates that there is no limit to the number of archives. Value 0 or a value less than -1 indicates that the HistoryServer throws "IllegalConfigurationException".

    -1

    -1

    historyserver.log.jobmanager.url-pattern

    JobManager log URL where you can be redirected to. A log archiving service is required as Flink does not collect logs for completed jobs.

    -

    -

    historyserver.log.taskmanager.url-pattern

    TaskManager log URL where you can be redirected to. A log archiving service is required as Flink does not collect logs for completed jobs.

    -

    -

    historyserver.web.refresh-interval

    Interval for refreshing the HistoryServer web UI, in milliseconds.

    10000

    10000

    historyserver.web.ssl.enabled

    Whether to enable HTTP access to the HistoryServer web UI. This parameter is valid only when the global SSL flag security.ssl.enabled is set to true.

    false

    false

  8. In the /opt/hadoopclient/Flink/flink/bin directory, run the following command to start the HistoryServer.

    ./historyserver.sh start

  9. Check the logs to see if the HistoryServer has started.

    cd /opt/hadoopclient/Flink/flink/log

    Check the flink-root-historyserver-xxx.log log file. If there are no errors reported and you can see the message "Web frontend listening at [ip]:port," the HistoryServer has started.

  10. Log in to the HistoryServer web UI. Enter "http://IP address of the HistoryServer web UI:HistoryServer web UI port (for example "192.168.x.x:8082") into your browser's address bar to access the HistoryServer page.

In security mode, you need to modify the jobmanager.web.allow-access-address configuration in the Client installation directory/Flink/flink/conf/flink-conf.yaml file and add the IP address of the host that needs to access the HistoryServer page to enable access to the page.

HistoryServer APIs

You can use API requests to operate the HistoryServer. Submit a request as follows:

  • Format of an API call

    http://hostname:port/jobs/<Job ID>/Request type

  • Example of an API call using job ID 7684be6004e4e955c2a558a9bc463f65

    http://hostname:port/jobs/7684be6004e4e955c2a558a9bc463f65/exceptions

Available Request Types

Available APIs

  • /config
  • /jobs/overview
  • /jobs/<jobid>
  • /jobs/<jobid>/vertices
  • /jobs/<jobid>/config
  • /jobs/<jobid>/exceptions
  • /jobs/<jobid>/accumulators
  • /jobs/<jobid>/vertices/<vertexid>
  • /jobs/<jobid>/vertices/<vertexid>/subtasktimes
  • /jobs/<jobid>/vertices/<vertexid>/taskmanagers
  • /jobs/<jobid>/vertices/<vertexid>/accumulators
  • /jobs/<jobid>/vertices/<vertexid>/subtasks/accumulators
  • /jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>
  • /jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>
  • /jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>/accumulators
  • /jobs/<jobid>/plan
  • /jobs/<jobid>/jobmanager/config
  • /jobs/<jobid>/jobmanager/environment
  • /jobs/<jobid>/jobmanager/log-url
  • /jobs/<jobid>/taskmanagers/<taskmanagerid>/log-url