Creating a Kafka_SSL Datasource Authentication

Scenario

Create a Kafka_SSL datasource authentication on the DLI console to store the Kafka authentication information to DLI. This will allow you to access to Kafka instances without having to configure a username and password in SQL jobs.

• When Kerberos authentication is enabled for MRS Kafka but SSL authentication is disabled, create a Kerberos authentication. When creating a table, configure krb_auth_name to associate the datasource authentication.
• If Kerberos authentication and SSL authentication are both enabled for MRS Kafka, you need to create Kerberos and Kafka_SSL authentications. When creating a table, configure krb_auth_name and ssl_auth_name to associate the datasource authentications.
• Datasource authentication is not required when Kerberos authentication is disabled but SASL authentication is enabled for MRS Kafka (for example, when a username and a password are used for PlainLoginModule authentication).
• When Kerberos authentication is disabled but SSL authentication is enabled for MRS Kafka, you need to create a Kafka_SSL authentication. When creating a table, configure ssl_auth_name to associate the datasource authentication.
• When Kerberos authentication is disabled but SASL authentication and SSL authentication are enabled for MRS Kafka, you need to create a Kafka_SSL authentication. When creating a table, configure ssl_auth_name to associate the datasource authentication.

Data Sources Supported by Kafka_SSL Datasource Authentication

Table 1 lists the data sources supported by Kafka_SSL datasource authentication.

Procedure

1. Download the authentication credential.
   • DMS Kafka
     1. Log in to the DMS (for Kafka) console and click a Kafka instance to access its details page.
     2. In the connection information, find the SSL certificate and click Download.

        Decompress the downloaded kafka-certs package to obtain the client.jks and phy_ca.crt files.

   • MRS Kafka
     1. Log in to MRS Manager.
     2. Choose System > Permission > User.
     3. Click More, select Download Authentication Credential, save the file, and decompress it to obtain the truststore file.
2. Upload the authentication credential to the OBS bucket.

3. Create a datasource authentication.
   1. Log in to the DLI management console.
   2. Choose Datasource Connections. On the page displayed, click Datasource Authentication.
   3. Click Create.

      Configure Kafka authentication parameters according to Table 2.

      Table 2 Parameters

      Parameter	Description
      Type	Select Kafka_SSL.
      Authentication Certificate	Name of the datasource authentication to be created. The name can contain only digits, letters, and underscores (_), but cannot contain only digits or start with an underscore (_). The name can contain a maximum of 128 characters.
      Truststore Path	OBS path to which the SSL truststore file is uploaded. For MRS Kafka, enter the OBS path of the Truststore.jks file. For DMS Kafka, enter the OBS path of the client.jks file.
      Truststore Password	Truststore password.
      Keystore Path	OBS path to which the SSL keystore file (key and certificate) is uploaded.
      Keystore Password	Keystore (key and certificate) password.
      Key Password	Password of the private key in the keystore file.

      Figure 1 Creating a datasource authentication – Kafka_SSL
      
4. Access Kafka with SASL_SSL authentication enabled.

   When creating a data source, associate the data source with the created datasource authentication to access the data source.

   Table 3 lists the fields used to associate with the datasource authentication during table creation.

   Table 3 Fields that are used to associate with Kafka_SSL datasource authentication during table creation

   Parameter	Mandatory	Data Type	Description
   ssl_auth_name	No	String	This field is used to associate datasource authentications when source, result, and dimension tables are created. Name of the created Kafka_SSL datasource authentication. This configuration is used when SSL is configured for Kafka. If only SSL is used, configure the following parameter: 'properties.security.protocol '= 'SSL'; If SASL_SSL is used, configure the following parameters: 'properties.security.protocol' = 'SASL_SSL', 'properties.sasl.mechanism' ='GSSAPI or PLAIN' 'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxx\" password=\"xxx\";'

   For details about how to create a table, see Data Lake Insight Syntax Reference.

   • Flink OpenSource SQL Syntax Reference: Creating a Kafka Source Table