What Is MRS?

Product Architecture

Figure 1 shows the MRS logical architecture.

MRS 3.x or later does not support patch management on the management console.

Figure 1 MRS architecture

MRS architecture includes infrastructure and big data processing phases.

• Infrastructure
  MRS big data clusters are built based on Elastic Cloud Server (ECS), and make full use of the high reliability and security capabilities of the virtualization layer.

  • A Virtual Private Cloud (VPC) is a virtual internal network provided for each tenant. It is isolated from other networks by default.
  • Elastic Volume Service (EVS) provides highly reliable and high-performance storage.
  • ECS provides scalable VMs, and works with VPCs, security groups, and the EVS multi-replica mechanism to build an efficient, reliable, and secure computing environment.
• Data collection

  The data collection layer provides the capability of importing data from various dta sources, such as Flume (data ingestion), Loader (relational data import), and Kafka (highly reliable message queue), to MRS big data clusters. Alternatively, you can use Cloud Data Migration (CDM) to import external data to MRS clusters.

• Data storage

  MRS clusters can store structured and unstructured data, and support multiple efficient formats to meet the requirements of different computing engines.

  • HDFS is a general-purpose distributed file system on a big data platform.
  • OBS is an object storage service that features high availability and low cost.
  • HBase supports data storage with indexes, and is applicable to high-performance index-based query scenarios.
• Data convergence processing
  • MRS provides multiple mainstream compute engines, including MapReduce (batch processing), Tez (DAG model), Spark (in-memory computing), Spark Streaming (micro-batch stream computing), Storm (stream computing), and Flink (stream computing), to convert data structures and logic into data models that meet service requirements in a variety of big data application scenarios.
  • Based on the preset data model and easy-to-use SQL data analysis, users can select Hive (data warehouse), SparkSQL, and Presto (interactive query engine).
• Data display and scheduling

  Displays data analysis results and integrates with Data Lake Governance Center (DGC) to provide a one-stop big data collaborative development platform, helping you easily complete multiple tasks, such as data modeling, data integration, script development, job scheduling, and O&M monitoring, making big data more accessible than ever before, and helping you effortlessly build big data processing centers.

• Cluster management

  All components of the Hadoop-based big data ecosystem are deployed in distributed mode, and their deployment, management, and O&M are complex.

  MRS provides a unified O&M management platform for cluster management, supporting one-click cluster deployment, multi-version selection, as well as manual scaling and auto scaling of clusters without service interruption. In addition, MRS provides job management, resource tag management, and O&M of the preceding data processing components at each layer. It also provides one-stop O&M capabilities, covering monitoring, alarm reporting, configuration, and patch upgrade.

Product Advantages

MRS has a powerful Hadoop kernel team and is deployed based on enterprise-level FusionInsight big data platform. MRS has been deployed on tens of thousands of nodes and can ensure Service Level Agreements (SLAs) for multi-level users.

MRS has the following advantages:

• High performance

  MRS supports self-developed CarbonData storage technology. CarbonData is a high-performance big data storage solution. It allows one data set to apply to multiple scenarios and supports features, such as multi-level indexing, dictionary encoding, pre-aggregation, dynamic partitioning, and quasi-real-time data query. This improves I/O scanning and computing performance and returns analysis results of tens of billions of data records in seconds. In addition, MRS supports self-developed enhanced scheduler Superior, which breaks the scale bottleneck of a single cluster and is capable of scheduling over 10,000 nodes in a cluster.

• Cost-effectiveness

  Based on diversified cloud infrastructure, MRS provides various computing and storage choices and separates computing from storage, delivering cost-effective massive data storage solutions. MRS supports auto scaling to address peak and off-peak service loads, releasing idle resources on the big data platform for customers. MRS clusters can be created and scaled out when you need them, and can be terminated or scaled in after you use them, minimizing cost.

• High security

  MRS delivers enterprise-level big data multi-tenant permissions management and security management to support table-based and column-based access control and data encryption.

• Easy O&M

  MRS provides a visualized big data cluster management platform, improving O&M efficiency. MRS supports rolling patch upgrade and provides visualized patch release information and one-click patch installation without manual intervention, ensuring long-term stability of user clusters.

• High reliability

  The proven large-scale reliability and long-term stability of MRS meet enterprise-level high reliability requirements. In addition, MRS supports automatic data backup across AZs and regions, as well as automatic anti-affinity. It allows VMs to be distributed on different physical machines.

Using MRS for the First Time

If you are a first-time user, get familiar with the following information:

• Basic concepts

  See Components to learn the basic knowledge of MRS, including the basic principles and enhanced features of each MRS component, as well as the unique concepts and functions of MRS.

• Getting started

  See Getting Started to learn how to use MRS. "Getting Started" provides detailed operation guidance of samples. You can create and use MRS clusters based on the operation guidance.

• Other functions and operation guides

  If you are an MRS cluster user and O&M engineer, you can perform operations such as cluster life cycle management, scaling, and job management by referring to Users Guide. See Component Operation Guide to learn how to use components in a cluster.

  If you are a developer, you can refer to APIs to manage MRS clusters and execute jobs. For details, see API Reference.