Huawei Cloud Big Data Components

The following lists common Huawei Cloud big data service components. You can refer to these components when designing the big data deployment architecture.

• MapReduce Service (MRS)

  With MRS, you can deploy your Hadoop clusters with just a few clicks and manage them on Huawei Cloud. MRS is fully compatible with open-source APIs and can easily run big data components such as Hadoop, Spark, HBase, Kafka, and Storm. MRS can be customized based on business requirements, helping enterprises quickly build a massive data processing system.

  For details, see MapReduce Service Documentation.

• Data Lake Insight (DLI)

  DLI is a serverless big data compute and analysis service that is fully compatible with Apache Spark, Apache Flink, and Trino. It provides streaming, batch, and interactive data processing. DLI supports standard SQL and is compatible with Spark SQL and Flink SQL. It also supports multiple access modes, and is compatible with mainstream data formats. Data on cloud-based platforms such as CloudTable, RDS, GaussDB (DWS), CSS, OBS, ECS databases, as well as offline databases, can be explored using SQL or programs. This eliminates the need for complex ETL processes.

  For details, see Data Lake Insight Documentation.

• Cloud Search Service (CSS)

  CSS is a distributed search engine service based on Elasticsearch, fully hosted on Huawei Cloud. You can use it for structured and unstructured data search, and use AI vectors for combine search, statistics, and reports. Elasticsearch is an open-source distributed search engine that can be deployed in standalone or cluster mode. As the heart of the ELK Stack, Elasticsearch clusters support multi-condition search, statistical analysis, and create visualized reports of structured and unstructured text.

  For details, see Cloud Search Service Documentation.

• GaussDB(DWS)

  GaussDB(DWS) is a native cloud service based on Huawei converged data warehouse GaussDB. It is compatible with standard ANSI SQL-99, SQL:2003, PostgreSQL, and Oracle database ecosystems. GaussDB(DWS) comes in three types: standard data warehouse, stream data warehouse, and hybrid data warehouse.

  For details, see GaussDB(DWS) Service Documentation.

• DataArts Studio

  DataArts Studio can be interconnected with all Huawei Cloud data lake and database services which function as the data lake foundation, such as MRS Hive and GaussDB(DWS). It can also be interconnected with traditional data warehouses, such as Oracle and MySQL.

  For details, see DataArts Studio Service Documentation.

• Data Ingestion Service (DIS)

  DIS addresses the challenge of transmitting data from outside the cloud to inside the cloud. It builds data streams for custom applications capable of processing or analyzing streaming data. DIS continuously captures, transmits, and stores terabytes of data from hundreds of thousands of sources every hour, such as logs, Internet of Things (IoT) data, social media feeds, website clickstreams, and location-tracking events.

  For details, see Data Ingestion Service Documentation.

• Cloud Data Migration (CDM)

  CDM is an efficient and easy-to-use data migration service. Based on big data migration to the cloud and the intelligent data lake solution, CDM provides easy-to-use migration capabilities and can integrate a broad set of data sources into the data lake, making data migration and integration easier and more efficient.

  For details, see Cloud Data Migration Service Documentation.

• Data Express Service (DES)

  DES is a transmission service for moving TB-level or hundreds of TB-level data to the cloud. DES allows you to transmit data by Teleport or by disk. Use disk mode for migrating data under 30 TB. Use Teleport mode for data ranging from 30 TB to 500 TB. For data over 500 TB, use Direct Connect.

  For details, see Data Express Service Documentation.