Help Center/ MapReduce Service/ User Guide/ Installing an MRS Cluster Client/ Installing a Client (MRS 3.x)
Updated on 2024-10-25 GMT+08:00
View PDF
Share

Installing a Client (MRS 3.x)

You need to install a cluster client to connect to the component server in the cluster and perform tasks such as component connection and job submission. You can install the cluster client on a node within the cluster or on a node outside of it.

To ensure that certain features function properly, it is important to reinstall the client after modifying the server configuration of a component in the cluster. This ensures that the client version matches the server version.

This section describes how to install the MRS 3.x cluster client. For MRS 2.x and earlier versions, see Installing a Client (MRS 2.x or Earlier). If the cluster contains the Flume component, install the Flume client separately. For details, see Installing the Flume Client.

Video Tutorial

This tutorial uses MRS 3.1.0 as an example to describe how to install and use a client.

The UI may vary depending on the version. This tutorial is for reference only.

Prerequisites

  • If the node where the client is to be installed is outside the cluster, the node must be able to communicate with the nodes in the MRS cluster. Otherwise, client installation will fail.
  • The node where the client is to be installed must have the NTP service enabled and synchronized time with the MRS cluster server. Otherwise, client installation will fail.
  • You can install the client on the node as a root or any OS user. The user must have the operation permission on the client file storage directory and installation directory.
  • When you install the client as a user other than omm or root, and the /var/tmp/patch directory already exists, you have changed the permission for the directory to 777 and changed the permission for the logs in the directory to 666.

Installing a Client on a Node in a Cluster

  1. Obtain the client software package.

    Log in to Manager by referring to Accessing MRS Manager. Click Cluster. On the Dashboard page, click the more sign (...) and select Download Client. In the displayed Download Cluster Client dialog box, configure parameters and click OK.
    Figure 1 Downloading a client
    • The client software package downloaded from the FusionInsight Manager homepage contains the clients of all services (excluding Flume) in the cluster. To download the client of a single service, choose Cluster > Services > Service name, click More, and select Download Client.
    • For MRS 3.3.0 or later, click Download Client on the Manager homepage.
    Table 1 Client download parameters

    Parameter

    Description

    Example Value

    Select Client Type
    • Complete Client: contains the complete client software package and configuration files. Generally, this option is selected.
    • Configuration Files Only: downloads only client configuration files in the scenario where the administrator modifies the component server configuration on FusionInsight Manager after the complete client is downloaded and installed in an application development task, and developers need to update client configuration files.

    Complete Client

    Select Platform Type

    The client type must match the architecture of the node where the client is to be installed. Otherwise, the installation fails.

    For clusters of the LTS version, only the client software package whose type is the same as that of FusionInsight Manager can be downloaded.

    • x86_64: indicates the client software package that can be deployed on a x86 platform.
    • aarch64: indicates the client software package that can be deployed on a Kunpeng server.

    x86_64

    Save to Path

    The path for storing the client software package on the active OMS node

    • Select Save to Path: Customize the path for storing the client software package on the active OMS node. User omm must have the read, write, and execute permissions on the path.

      If the path is not changed, the client file generated is saved in the /tmp/FusionInsight-Client directory on the active OMS node in the cluster by default.

    • Not to select Save to Path: The generated client file is automatically downloaded and saved to the local host. Before installing the client, you need to upload the file to a specified directory on the target node.

    Select Save to Path

  2. Copy the client software package to a specified directory on the node where the client is to be installed.

    By default, the client software package is stored on the active OMS node in the cluster. To install the client on other nodes in the cluster, log in to the active OMS node as user omm and run the following command to copy the software package to the specified node. Otherwise, skip this step.

    For example, copy the software package to the /tmp/clienttemp directory.

    scp -p /tmp/FusionInsight-Client/FusionInsight_Cluster_1_Services_Client.tar IP address of the node where the client is to be installed:/tmp/clienttemp

  3. Log in to the node where the client is to be installed as the client installation user.

    You can install the client on the node as user root, omm, or any other OS user. The user needs to have the operation permission on the directory for storing the client file and the installation directory. The permission on the two directories is 755.

  4. Decompress the client software package.

    1. Go to the directory where the package is stored, for example, /tmp/clienttemp.

      cd /tmp/clienttemp

    2. Decompress the installation package.

      tar -xvf FusionInsight_Cluster_1_Services_Client.tar

    3. Run the sha256sum command to verify the decompressed file and check whether the command output is consistent with the content in the sha256 file.

      sha256sum -c FusionInsight_Cluster_1_Services_ClientConfig.tar.sha256

      FusionInsight_Cluster_1_Services_ClientConfig.tar: OK
    4. Decompress the installation package.

      tar -xvf FusionInsight_Cluster_1_Services_ClientConfig.tar

  5. Go to the directory where the client software package is decompressed and run the following command to install the client to the specified directory:

    cd FusionInsight_Cluster_1_Services_ClientConfig

    ./install.sh Client installation directory

    Example:

    ./install.sh /opt/hadoopclient

    Wait until the client installation is complete.

    ...
The component client is installed successfully
    • When installing the client, you can choose the installation directory. If you opt to use an existing directory, it must be empty. Also, the installation directory cannot have spaces and should only contain uppercase and lowercase letters, digits, and underscores (_).
    • Delete the client installation directory when uninstalling a client.
    • To ensure that the client can be used only by the current user, add the -o parameter during the installation. For example, run the ./install.sh /opt/hadoopclient -o command to install the client.

  6. Check whether the client is installed.

    1. Go to the client installation directory and load environment variables.

      cd /opt/hadoopclient

      source bigdata_env

    2. Run related commands based on the cluster mode.
      • If Kerberos authentication is not enabled for the cluster, you can directly run commands related to the component client.

        View files in the HDFS root directory:

        hdfs dfs -ls /

      • If Kerberos authentication is enabled for the cluster, run the kinit command to perform user authentication.

        Example:

        kinit admin

        Password for xxx@HADOOP.COM: #Enter the password of user admin.
        Run the klist command to query and confirm authentication details. 
        Ticket cache: FILE:/tmp/krb5cc_0 
Default principal: xxx@HADOOP.COM   

Valid starting       Expires              Service principal 
...

Installing a Client on a Node Outside a Cluster

  1. Prepare a Linux ECS for installing the MRS cluster client.

    • The recommended ECS OSs and versions are as follows:
      Table 2 OS reference list

      CPU Architecture

      OS

      Supported Version

      x86 computing

      EulerOS

      EulerOS 2.5

      SUSE

      SUSE Linux Enterprise Server 12 SP4 (SUSE 12.4)

      Red Hat

      Red Hat-7.5-x86_64 (Red Hat 7.5)

      CentOS

      CentOS 7.6

      Kunpeng computing

      EulerOS

      EulerOS 2.8

      CentOS

      CentOS 7.6
    • Sufficient disk space (at least 40 GB) must be allocated to the ECS client installation directory.
    • The ECS must be in the same VPC and security group as the MRS cluster.
    • All ports in the inbound direction of the MRS cluster security group are open to the client node. For details, see Adding a Security Group Rule.
    • The NTP service has been installed on the ECS OS and is running properly.

      If the NTP service is not installed, run the yum install ntp -y command to install it when the yum source is configured.

    • The ECS must allow users to log in to it using a password (SSH).

  2. Configure NTP time synchronization for the node where the client is to be installed to synchronize time with the MRS cluster.

    1. Log in to the MRS console and click the MRS cluster name from the cluster list.
    2. Click Nodes to expand the Master node group list and view the IP addresses of the Master1 and Master2 nodes in the cluster.
      Figure 2 Viewing the IP address of the Master node
    3. Log in to the node where the client is to be installed as user root and run the following command to edit the NTP configuration file. Add the IP addresses of the Master1 and Master2 nodes in the MRS cluster and comment out other server addresses.
      vi /etc/ntp.conf 
      server master1_ip prefer
server master2_ip
      Figure 3 Modifying the NTP configuration file
    4. Save the configuration file and run the following command to disable the NTP service:

      service ntpd stop

    5. Manually synchronize the time.

      /usr/sbin/ntpdate IP address of the active Master node

    6. Start the NTP service.

      service ntpd start

      Or

      systemctl restart ntpd

    7. Run the ntpstat command to check the time synchronization result. 
      synchronised to NTP server (xxx) at stratum 2
   time correct to within 12 ms
   polling server every 16 s

  3. Obtain the client software package.

    Log in to Manager by referring to Accessing MRS Manager. Click Cluster. On the Dashboard page, click the more sign (...) and select Download Client. In the displayed Download Cluster Client dialog box, configure parameters and click OK.
    Figure 4 Downloading a client
    • The client software package downloaded from the FusionInsight Manager homepage contains the clients of all services (excluding Flume) in the cluster. To download the client of a single service, choose Cluster > Services > Service name, click More, and select Download Client.
    • For MRS 3.3.0 or later, click Download Client on the Manager homepage.
    Table 3 Client download parameters

    Parameter

    Description

    Example Value

    Select Client Type
    • Complete Client: contains the complete client software package and configuration files. Generally, this option is selected.
    • Configuration Files Only: downloads only client configuration files in the scenario where the administrator modifies the component server configuration on FusionInsight Manager after the complete client is downloaded and installed in an application development task, and developers need to update client configuration files.

    Complete Client

    Select Platform Type

    The client type must match the architecture of the node where the client is to be installed. Otherwise, the installation fails.

    For clusters of the LTS version, only the client software package whose type is the same as that of FusionInsight Manager can be downloaded.

    • x86_64: indicates the client software package that can be deployed on a x86 platform.
    • aarch64: indicates the client software package that can be deployed on a Kunpeng server.

    x86_64

    Save to Path

    The path for storing the client software package on the active OMS node

    • Select Save to Path: Customize the path for storing the client software package on the active OMS node. User omm must have the read, write, and execute permissions on the path.

      If the path is not changed, the client file generated is saved in the /tmp/FusionInsight-Client directory on the active OMS node in the cluster by default.

    • Not to select Save to Path: The generated client file is automatically downloaded and saved to the local host. Before installing the client, you need to upload the file to a specified directory on the target node.

    Select Save to Path

  4. Copy the client software package to a specified directory on the node where the client is to be installed.

    By default, the client software package is stored on the active OMS node in the cluster. To install the client on other nodes in the cluster, log in to the active OMS node as user omm and run the following command to copy the software package to the specified node. Otherwise, skip this step.

    For example, copy the software package to the /tmp/clienttemp directory.

    scp -p /tmp/FusionInsight-Client/FusionInsight_Cluster_1_Services_Client.tar IP address of the node where the client is to be installed:/tmp/clienttemp

  5. Log in to the node where the client is to be installed as the client installation user.

    You can install the client on the node as user root, omm, or any other OS user. The user needs to have the operation permission on the directory for storing the client file and the installation directory. The permission on the two directories is 755.

  6. Decompress the client software package.

    1. Go to the directory where the package is stored, for example, /tmp/clienttemp.

      cd /tmp/clienttemp

    2. Decompress the installation package.

      tar -xvf FusionInsight_Cluster_1_Services_Client.tar

    3. Run the sha256sum command to verify the decompressed file and check whether the command output is consistent with the content in the sha256 file.

      sha256sum -c FusionInsight_Cluster_1_Services_ClientConfig.tar.sha256

      FusionInsight_Cluster_1_Services_ClientConfig.tar: OK
    4. Decompress the installation package.

      tar -xvf FusionInsight_Cluster_1_Services_ClientConfig.tar

  7. Check the network connection of the client.

    1. To install the client on a node, map the host names and IP addresses of all nodes in the cluster in the /etc/hosts file. Here are the steps to import the domain name mapping of the cluster to the hosts file:
      1. Switch to user root or a user who has the permission to modify the hosts file.

        su - root

      2. Go to the directory where the client package is decompressed.

        cd /tmp/clienttemp/FusionInsight_Cluster_1_Services_ClientConfig

      3. Import the domain name mapping to the hosts file.

        cat realm.ini >> /etc/hosts

    • If the host where the client is installed is not a node in the cluster, configure network connections for the client to prevent errors when you run commands on the client.
    • If Spark tasks are executed in yarn-client mode, add the spark.driver.host parameter to the file Client installation directory/Spark/spark/conf/spark-defaults.conf and set the parameter to the client IP address.
    • If the yarn-client mode is used, you need to configure the mapping between the IP address and host name of the client in the hosts file on the active and standby YARN nodes (ResourceManager nodes in the cluster) to make sure that the Spark web UI is properly displayed.

  8. Go to the directory where the client software package is decompressed and run the following command to install the client to the specified directory:

    cd FusionInsight_Cluster_1_Services_ClientConfig

    ./install.sh Client installation directory

    Example:

    ./install.sh /opt/hadoopclient

    Wait until the client installation is complete.

    ...
The component client is installed successfully
    • When installing the client, you can choose the installation directory. If you opt to use an existing directory, it must be empty. Also, the installation directory cannot have spaces and should only contain uppercase and lowercase letters, digits, and underscores (_).
    • Delete the client installation directory when uninstalling a client.
    • To ensure that the client can be used only by the current user, add the -o parameter during the installation. For example, run the ./install.sh /opt/hadoopclient -o command to install the client.

  9. Check whether the client is installed.

    1. Go to the client installation directory and load environment variables.

      cd /opt/hadoopclient

      source bigdata_env

    2. Run related commands based on the cluster mode.
      • If Kerberos authentication is not enabled for the cluster, you can directly run commands related to the component client.

        View files in the HDFS root directory:

        hdfs dfs -ls /

      • If Kerberos authentication is enabled for the cluster, run the kinit command to perform user authentication.

        Example:

        kinit admin

        Password for xxx@HADOOP.COM: #Enter the password of user admin.
        Run the klist command to query and confirm authentication details. 
        Ticket cache: FILE:/tmp/krb5cc_0 
Default principal: xxx@HADOOP.COM   

Valid starting       Expires              Service principal 
...