Interconnecting Flink with AOM

Updated on 2024-12-13 GMT+08:00

View PDF

This topic is available for MRS 3.5.0 and later versions only.

Scenarios

Application Operations Management (AOM) is an an observable platform and provides integrated monitoring capabilities based on metrics, links, logs, and events. Flink can send monitoring indicators to AOM via Prometheus instances, making it easy to view monitored data.

This example describes how to interconnect Flink with AOM using FlinkResource, FlinkServer, or a client and upload Flink monitoring metrics to AOM.

Prerequisites

AOM has been enabled.
The HDFS, YARN, Kafka, and Flink components have been installed in the MRS cluster.
The client that contains the Flink service has been installed, for example, in the /opt/client directory.

Procedure

Create a Prometheus instance in AOM.

Log in to the AOM console.
In the navigation pane on the left, choose Prometheus Monitoring > Instances. On the displayed page, click Add Prometheus Instance.

Enter an instance name, specify the enterprise project and instance type, and click OK.

**Table 1** Parameters for creating a Prometheus instance
Parameter	Description
Instance Name	Prometheus instance name. Enter a maximum of 100 characters and do not start or end with an underscore (_) or hyphen (-). Only letters, digits, underscores, and hyphens are allowed.
Enterprise Project	Enterprise project name. If you have selected ALL for Enterprise Project on the global settings page, select an enterprise project from the drop-down list. If you have already selected an enterprise project on the global settings page, this option will be grayed and cannot be changed.
Instance Type	Type of the Prometheus instance. Select Common Prometheus Instance.

In the instance list, click the name of the created Prometheus instance you just created. Click Settings in the navigation pane, and check and save the following information.

**Table 2** Prometheus instance information
Type	Parameter	Example Data
Credential	AppSecret	0000000100000001070C6Axxx0E2EF73
Service Address	URL in the Configuration Code for Prometheus Remote Write pane in the Intranet tab	https://aom-internal-access.xxx/push

Configure parameters for interconnecting with AOM on MRS.

Log in to FusionInsight Manager.
Choose Cluster > Services > Flink > Configurations > All Configurations.

Configure parameters for Flink to interconnect with AOM.

Interconnect with AOM through FlinkServer.

Choose FlinkServer(Role) > Customization, add parameters in Table 3 to flink.customized.configs, save the configuration, and restart involved FlinkServer instances.

**Table 3** Parameters for interconnecting Flink with AOM
Parameter Name	Value	Mandatory	Description
metrics.reporter	alarm,aom	Yes	alarm means FlinkServer alarms for customized parameters overwriting original Flink parameters. Set both alarm and aom.
metrics.reporter.aom.url	URL obtained from 1.d	Yes	URL for connecting to the AOM Prometheus instance.
metrics.reporter.aom.access.code	Value of AppSecret obtained from 1.d	Yes	Credential for calling the AOM Prometheus instance.
metrics.reporter.aom.factory.class	com.huawei.mrs.flink.AomMetricReporterFactroy	Yes	Implementation class for reporting Flink metrics. The fixed value is com.huawei.mrs.flink.AomMetricReporterFactroy
metrics.reporter.aom.interval	30s	Yes	Interval for reporting Flink metrics, in seconds.
metrics.reporter.aom.version	0.1.0	No	AOM version information.
metrics.reporter.aom.filterLabelValueCharacters	true	No	Whether to filter the label value. The content that cannot be matched by [a-zA-Z0-9:_] will be filtered. true (default value): Enable the filtering function. false: Disable the filtering function.
metrics.reporter.aom.jobName	-	No	Name of the job. The name will be sent to the Prometheus instance as a label value.
metrics.reporter.aom.randomJobNameSuffix	true	No	Whether to add a random suffix to the job name. true (default value): Add a random suffix. false: Do not add a random suffix.
metrics.reporter.aom.groupingKey	-	No	Grouping key, which is the group and global label of all metrics. The value is in the format of k1=v1,k2=v2.

Interconnect with AOM through FlinkResource.
1. Choose FlinkResource(Role) > Customization, add parameters in Table 3 to flink.customized.configs, save the configuration, and restart involved FlinkResource instances.
2. Reinstall the client or update the existing client configuration.
Interconnect with AOM through the Flink client.
1. Log in to the node where the client is installed as the client installation user.
2. Add the parameters in Table 3 to the /opt/hadoopclient/Flink/flink/conf/flink-conf.yaml file and save the file.

View monitoring metrics.

After the Flink job is successfully executed, log in to the AOM 2.0 console. In the navigation pane on the left, choose Prometheus Monitoring > Instances. Click the name of the Prometheus instance created in 1. View the monitoring information on the Metric Management page.