Help Center/ MapReduce Service/ User Guide (Paris Region)/ Configuring a Cluster/ Quick Creation of a Cluster/ Quick Creation of a Hadoop Analysis Cluster
Updated on 2024-10-11 GMT+08:00
View PDF
Share

Quick Creation of a Hadoop Analysis Cluster

This section describes how to quickly create a Hadoop analysis cluster for analyzing and querying vast amounts of data. In the open-source Hadoop ecosystem, Hadoop uses Yarn to manage cluster resources, Hive and Spark to provide offline storage and computing of large-scale distributed data, Spark Streaming and Flink to offer streaming data computing, and Presto to enable interactive queries, Tez to provide a distributed computing framework of directed acyclic graphs (DAGs).

The Hadoop analysis cluster consists of the following components:

  • MRS 1.8.9: Hadoop 2.8.3, Spark 2.2.1, Hive 1.2.1, Presto 0.215, and Flink 1.7.0.
  • MRS 2.0.1: Hadoop 3.1.1, Spark 2.3.2, Hive 3.1.0, Presto 308, Tez 0.9.1, and Flink 1.7.0.
  • MRS 3.1.0-LTS.1: Hadoop 3.1.1, Hive 3.1.0, Spark2x 2.4.5, Flink 1.12.0, ZooKeeper 3.5.6, Ranger 2.0.0 and Tez 0.9.2.
  • MRS 3.1.2-LTS.3: Hadoop 3.1.1, Hive 3.1.0, Spark2x 3.1.1, Flink 1.12.2, ZooKeeper 3.6.3, Ranger 2.0.0, and Tez 0.9.2.
  • MRS 3.2.0-LTS.1: Hadoop 3.3.1, Hive 3.1.0, Spark2x 3.1.1, Flink 1.15.0, ZooKeeper 3.6.3, Ranger 2.0.0 and Tez 0.9.2.

Quick Creation of a Hadoop Analysis Cluster

  1. Log in to the MRS console.
  2. Click Create Cluster. The page for creating a cluster is displayed.
  3. Click the Quick Config tab.
  4. Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.

    • Region: Use the default value.
    • Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
    • Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
    • Component: Select Hadoop analysis cluster.
    • AZ: Use the default value.
    • VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
    • Subnet: Use the default value.
    • Enterprise Project: Use the default value.
    • CPU Architecture: Use the default value. This parameter is available in MRS 3.2.0-LTS.1 or later.
    • Cluster Node: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
    • Cluster HA: Use the default value. This parameter is not available in MRS 3.x.
    • Kerberos Authentication: Select whether to enable Kerberos authentication.
    • Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page. User root is supported in MRS 3.2.0-LTS.1 or later.
    • Password: Set a password for user root/admin.
    • Confirm Password: Enter the password of user root/admin again.
    • Key Pair: Select a key pair from the drop-down list to log in to an ECS. Select "I acknowledge that I have obtained private key file SSHkey-xxx and that without this file I will not be able to log in to my ECS." If you have never created a key pair, click View Key Pair to create or import a key pair. And then, obtain a private key file. This parameter is available in MRS 3.1.2-LTS.3 orearlier.

  5. Select Enable to enable secure communications. For details, see Communication Security Authorization.
  6. Click Create Now.

    If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.

  7. Click Back to Cluster List to view the cluster status. Click Access Cluster to view cluster details.

    For details about cluster status during creation, see the description of the status parameters in Table 1.

    It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.

    On the MRS management console, a maximum of 10 clusters can be concurrently created, and a maximum of 100 clusters can be managed.