Connecting the Open-Source confluent-kafka-go to the Security Cluster of MRS

Issue

How do I connect the Open-Source confluent-kafka-go an MRS security cluster?

Symptom

The open-source confluent-kafka-go fails to connect to the security cluster of MRS.

Cause Analysis

By default, the librdkafka library on which confluent-kafka-go depends uses the hostname of the broker as a part of the server principle. As a result, the authentication fails.

Procedure

The procedure for modifying the librdkafka is as follows:

1. The librdkafka source code address: https://github.com/edenhill/librdkafka
2. Add the sasl.kerberos.service.name configuration item to the src/rdkafka_conf.c file.

    "Kerberos principal name that Kafka runs as.",            .sdef = "kafka" },          { _RK_GLOBAL, "sasl.kerberos.principal", _RK_C_STR,            _RK(sasl.principal),            "This client´s Kerberos principal name.",            .sdef = "kafkaclient" }, +        { _RK_GLOBAL, "sasl.kerberos.domain.name", _RK_C_STR, +          _RK(sasl.domain_name), +          "This cluster´s Kerberos domain name.", +          .sdef = "hadoop.hadoop.com" },   #ifndef _MSC_VER          { _RK_GLOBAL, "sasl.kerberos.kinit.cmd", _RK_C_STR,            _RK(sasl.kinit_cmd),            "Full kerberos kinit command string, %{config.prop.name} is replaced "            "by corresponding config object value, %{broker.name} returns the "            "broker´s hostname.", -          .sdef = "kinit -S \"%{sasl.kerberos.service.name}/%{broker.name}\" " +          .sdef = "kinit -S \"%{sasl.kerberos.service.name}/%{sasl.kerberos.domain.name}\" "            "-k -t \"%{sasl.kerberos.keytab}\" %{sasl.kerberos.principal}" },          { _RK_GLOBAL, "sasl.kerberos.keytab", _RK_C_STR,            _RK(sasl.keytab),            "Path to Kerberos keytab file. Uses system default if not set."            "**NOTE**: This is not automatically used but must be added to the "            "template in sasl.kerberos.kinit.cmd as "

3. Add the domain_name field to the src/rdkafka_conf.h file.

   --- src\rdkafka_conf.h        2017-10-17 11:20:56.000000000 +0800 +++ src\rdkafka_conf.h        2017-10-25 16:26:34.000000000 +0800 @@ -118,12 +118,13 @@          struct {                  const struct rd_kafka_sasl_provider *provider;                  char *principal;                  char *mechanisms;                  char *service_name; +                                char *domain_name;                  char *kinit_cmd;                  char *keytab;                  int   relogin_min_time;                  char *username;                  char *password; #if WITH_SASL_SCRAM

4. Replace hostname with domainName in the src/rdkafka_sasl_cyrus.c file.

   --- src\rdkafka_sasl.c        2017-10-17 11:20:56.000000000 +0800 +++ src\rdkafka_sasl.c        2017-10-25 16:09:38.000000000 +0800 @@ -192,13 +192,14 @@                              rk->rk_conf.sasl.mechanisms,                              rk->rk_conf.api_version_request ? "" :                              ": try api.version.request=true");                  return -1;          } -        rd_strdupa(&hostname, rktrans->rktrans_rkb->rkb_nodename); +        //rd_strdupa(&hostname, rktrans->rktrans_rkb->rkb_nodename); +                rd_strdupa(&hostname, rk->rk_conf.sasl.domain_name);          if ((t = strchr(hostname, ´:´)))                  *t = ´\0´;  /* remove ":port" */

5. Recompile librdkafka (ensure that libsasl2-dev has been installed). For details, see https://github.com/edenhill/librdkafka/tree/v0.11.1.

   ./configure make make install

6. Add the following configuration items when using the client:

   "security.protocol": "SASL_PLAINTEXT", 
   "sasl.kerberos.service.name": "kafka", 
   "sasl.kerberos.keytab": "/opt/nemon/user.keytab",  
   "sasl.kerberos.principal": "nemon@HADOOP.COM", 
   "sasl.kerberos.domain.name": "hadoop.hadoop.com",

   

   MRS 2.1.x and earlier versions:

   • sasl.kerberos.keytab: On MRS Manager, choose System > Manage User. Locate the row that contains the target user, choose More > Download authentication credential. Save the file and decompress it to extract the user.keytab file.
   • sasl.kerberos.principal: Enter the actual user name.
   • sasl.kerberos.domain.name: The domain naming rule is hadoop. toLowerCase(realm): If the cluster domain name (default_realm) is huawei.com, the value of domain is hadoop.huawei.com. On MRS Manager, choose Services > KrbServer > Service Configuration > All, and search for and view the value of default_realm.

   For MRS 3.x or later:

   • sasl.kerberos.keytab: On FusionInsight Manager, choose System > Permission > User. Locate the row that contains the target user and choose More >Download authentication credential from the Operation column. Save the file and decompress it to extract the user.keytab file.
   • sasl.kerberos.principal: Enter the actual user name.
   • sasl.kerberos.domain.name: The domain naming rule is hadoop. toLowerCase(realm): If the cluster domain name (default_realm) is huawei.com, the value of domain is hadoop.huawei.com. On FusionInsight Manager, choose Cluster > Service > KrbServer > > Configurations > All Configurations to search for and view the value of default_realm.