Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ Spark Development Guide/ FAQs About Spark Application Development/ How Do I Connect to Hive and HDFS of an MRS Cluster when the Spark Program Is Running on a Local Host?
Updated on 2024-08-16 GMT+08:00

How Do I Connect to Hive and HDFS of an MRS Cluster when the Spark Program Is Running on a Local Host?

Question

How do I connect to Hive and HDFS of an MRS cluster when the Spark program is running on a local host?

Answer

  1. Apply for and bind an elastic public IP address for each master node.
  2. Configure the mapping between the cluster IP addresses and host names on the local Windows host. Log in to the cluster background, run the cat /etc/hosts command, and copy the mapping between IP addresses and host names in the hosts file to C:\Windows\System32\drivers\etc\hosts. Host names are subject to the query result.

    192.168.0.90 node-master1BedB.089d8c43-12d5-410c-b980-c2728a305be3.com
    192.168.0.129 node-ana-corezLaR.089d8c43-12d5-410c-b980-c2728a305be3.com

  3. Log in to the background of any master node in the MRS cluster as user root and run the cat /etc/hosts command to obtain the mapping between IP addresses and host names in the hosts file.
  4. In C:\Windows\System32\drivers\etc\hosts of the local Windows host, configure the mapping obtained in 3 and change the IP addresses of all master nodes to the EIPs bound to the nodes.
  5. Save /opt/client/Hive/Beeline/conf/core-site.xml, /opt/client/Hive/config/hiveclient.properties, and /opt/client/Hive/config/hive-site.xml of the MRS cluster to the conf directory of the project.
  6. Log in to MRS Manager and choose System > Manage User.
  7. Select a user who has Hive privileges, choose More > Download Authentication Credential in the Operation column, save the file, and decompress the package to obtain the user.keytab and krb5.conf files.
  8. Change the IP address of the master node in the krb5.conf file to the EIP bound to the node. Save the user.keytab and krb5.conf files to the conf directory of the project.
  9. Modify the security group rules of the MRS cluster and change the IP address policy of the Windows where IDEA resides to allow all IP addresses.