MRS Data Source Usage Overview

MRS Cluster Overview

MRS is a big data cluster running based on the open-source Hadoop ecosystem. It provides the industry's latest cutting-edge storage and analysis capabilities of massive volumes of data, satisfying your data storage and processing requirements. For details about MRS, see the MapReduce Service User Guide.

You can use Hive/Spark (analysis cluster of MRS) to store massive volumes of service data. Hive/Spark data files are stored in HDFS. On DWS, you can connect a data warehouse cluster to MRS clusters, read data from HDFS files, and write the data to DWS when the clusters are on the same network.

Application Scenarios

DWS and MRS are good at processing different types of data.

• DWS: provides standard SQL queries and high-concurrency analytics for structured data and is suitable for data warehouses.
• MRS provides batch processing and analytics capabilities of components such as Hive and Spark for massive amounts of diversified data and is suitable for big data processing.

If you want to combine the service data processed by the Hive or Spark component of an MRS cluster and stored in HDFS with the data in a DWS cluster for unified analysis, you can configure an MRS data source connection to import data files from HDFS to DWS.

Example: You use the Spark component of an MRS cluster to clean and aggregate numerous service logs. The results are stored in HDFS. If you want to perform correlation analysis using the cleaned data and the order tables in DWS, you can use the MRS data source to import data from HDFS to DWS for unified query and analysis.

Notes and Constraints

• Multiple MRS data sources can exist on the same network, but one DWS cluster can connect to only one MRS cluster at a time.
• The DWS cluster and the MRS cluster must use the same network and can communicate with each other. Otherwise, the data source connection cannot be created or data cannot be imported.
• When the HDFS configuration of the MRS cluster changes, you need to update the MRS data source configuration in DWS in a timely manner. Otherwise, the established data source connection may become unavailable.

Operation Process

Perform the following operations to import data from MRS to a data warehouse cluster:

1. Create an MRS cluster. For details, see Buying a Custom Cluster.
2. Create an HDFS foreign table for querying data from the MRS cluster over APIs of a foreign server.

   For details, see Importing Data from MRS to a Data Warehouse Cluster in Data Warehouse Service (DWS) Data Migration and Synchronization.

3. In the DWS cluster, create an MRS data source connection according to Creating an MRS Data Source Connection.
4. Import data from an MRS data source to the cluster. For details, see Importing Data from MRS to a Cluster.
5. (Optional) When the HDFS configuration of the MRS cluster changes, update the MRS data source configuration in DWS. For details, see Updating the MRS Data Source Configuration.

Documentation

• MRS service introduction: What Is MRS?
• Best practice of importing data from MRS to DWS: Importing Table Data from MRS Hive to a DWS Cluster