Updated on 2024-09-30 GMT+08:00

Creating a Kafka_SSL Datasource Authentication

Scenario

Create a Kafka_SSL datasource authentication on the DLI console to store the Kafka authentication information to DLI. This will allow you to access to Kafka instances without having to configure a username and password in SQL jobs.

  • When Kerberos authentication is enabled for MRS Kafka but SSL authentication is disabled, create a Kerberos authentication. When creating a table, configure krb_auth_name to associate the datasource authentication.
  • If Kerberos authentication and SSL authentication are both enabled for MRS Kafka, you need to create Kerberos and Kafka_SSL authentications. When creating a table, configure krb_auth_name and ssl_auth_name to associate the datasource authentications.
  • Datasource authentication is not required when Kerberos authentication is disabled but SASL authentication is enabled for MRS Kafka (for example, when a username and a password are used for PlainLoginModule authentication).
  • When Kerberos authentication is disabled but SSL authentication is enabled for MRS Kafka, you need to create a Kafka_SSL authentication. When creating a table, configure ssl_auth_name to associate the datasource authentication.
  • When Kerberos authentication is disabled but SASL authentication and SSL authentication are enabled for MRS Kafka, you need to create a Kafka_SSL authentication. When creating a table, configure ssl_auth_name to associate the datasource authentication.

Data Sources Supported by Kafka_SSL Datasource Authentication

Table 1 lists the data sources supported by Kafka_SSL datasource authentication.

Table 1 Data sources supported by Kafka_SSL datasource authentication

Job Type

Table Type

Data Source

Notes and Constraints

Flink OpenSource SQL

Source table and result table

Kafka

SASL_SSL authentication has been enabled for DMS Kafka.

SASL authentication has been enabled for MRS Kafka.

SSL authentication has been enabled for MRS Kafka.

Procedure

  1. Download the authentication credential.
    • DMS Kafka
      1. Log in to the DMS (for Kafka) console and click a Kafka instance to access its details page.
      2. In the connection information, find the SSL certificate and click Download.

        Decompress the downloaded kafka-certs package to obtain the client.jks and phy_ca.crt files.

    • MRS Kafka
      1. Log in to MRS Manager.
      2. Choose System > Permission > User.
      3. Click More, select Download Authentication Credential, save the file, and decompress it to obtain the truststore file.
  2. Upload the authentication credential to the OBS bucket.

  3. Create a datasource authentication.
    1. Log in to the DLI management console.
    2. Choose Datasource Connections. On the page displayed, click Datasource Authentication.
    3. Click Create.

      Configure Kafka authentication parameters according to Table 2.

      Table 2 Parameters

      Parameter

      Description

      Type

      Select Kafka_SSL.

      Authentication Certificate

      Name of the datasource authentication to be created.

      • The name can contain only digits, letters, and underscores (_), but cannot contain only digits or start with an underscore (_).
      • The name can contain a maximum of 128 characters.

      Truststore Path

      OBS path to which the SSL truststore file is uploaded.

      • For MRS Kafka, enter the OBS path of the Truststore.jks file.
      • For DMS Kafka, enter the OBS path of the client.jks file.

      Truststore Password

      Truststore password.

      Keystore Path

      OBS path to which the SSL keystore file (key and certificate) is uploaded.

      Keystore Password

      Keystore (key and certificate) password.

      Key Password

      Password of the private key in the keystore file.

      Figure 1 Creating a datasource authentication – Kafka_SSL
  4. Access Kafka with SASL_SSL authentication enabled.

    When creating a data source, associate the data source with the created datasource authentication to access the data source.

    Table 3 lists the fields used to associate with the datasource authentication during table creation.

    Table 3 Fields that are used to associate with Kafka_SSL datasource authentication during table creation

    Parameter

    Mandatory

    Data Type

    Description

    ssl_auth_name

    No

    String

    This field is used to associate datasource authentications when source, result, and dimension tables are created.

    Name of the created Kafka_SSL datasource authentication. This configuration is used when SSL is configured for Kafka.

    • If only SSL is used, configure the following parameter:

      'properties.security.protocol '= 'SSL';

    • If SASL_SSL is used, configure the following parameters:
      • 'properties.security.protocol' = 'SASL_SSL',
      • 'properties.sasl.mechanism' ='GSSAPI or PLAIN'
      • 'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxx\" password=\"xxx\";'

    For details about how to create a table, see Data Lake Insight Syntax Reference.