Managing FlinkServer Catalogs

Scenarios

FlinkServer job development often requires accessing tables across various data sources, including Hive, Hudi, and Iceberg. Manually creating Flink SQL table definitions for each job is repetitive and risks schema inconsistencies with the underlying data source. Catalog management enables FlinkServer to automatically connect to underlying metadata, discover tables, and manage them centrally, streamlining job development.

Notes and Constraints

• This section applies to MRS 3.6.0-LTS.1 and later.
• You cannot change the catalog name after creating the catalog because Flink SQL statements use this name.
• You can only connect to the Hive MetaStore in this cluster. LakeFormation is not supported.
• By default, catalog management is enabled for new clusters. Cluster connections, data connections, and stream table management are not supported.
• Each application supports a maximum of 10 catalogs.
• In Flink SQL, you must use the three-part format to access tables in a catalog: `${catalog_name}`.`${db_name}`.`${table_name}`.
• The Flink engine does not verify permissions for databases and tables in catalogs.
• Catalog management in FlinkServer can be enabled only in active/standby mode.

Prerequisites

• Hive and Flink services are installed and running properly in the cluster.
• Catalog management in FlinkServer has been enabled by setting flinkserver.catalog.module.enabled to true and restarting the FlinkServer instance. This parameter is a role-level configuration. Changing its value updates the configuration across all instances.
  1. Log in to MRS Manager, choose Cluster > Services > Flink, and click Configurations and then All Configurations. Search for flinkserver.catalog.module.enabled, set it to true, click Save, and then click OK in the confirmation dialog box.

     For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

  2. Click the Instances tab, select all FlinkServer instances, and choose More > Restart Instance .

Creating a Catalog

1. Access the Flink web UI by referring to Accessing the FlinkServer Web UI.
2. Choose System Management > Catalog Management.
3. Click Create Catalog. On the displayed page, set parameters by referring to Table 1 and click OK.

   Table 1 Parameters required for creating a catalog

   Parameter	Description
   Name	The catalog name.
   Hive Configuration File	Hive configuration file, which is hive-site.xml. Method for obtaining the Hive configuration file: Log in to MRS Manager, choose Cluster > Services > Hive, click More in the upper right corner, and select Download Client. Set Select Client Type to Configuration Files Only, specify Select Platform Type and Select Download Location, and click OK.
   Type	Catalog type. Available options include Hive, Hudi, and Iceberg.
   Description	Description of the catalog.
   Customized Configuration	"Configure the catalog based on your specific service requirements.