Help Center> DataArts Studio> FAQs> Consultation and Billing> Database, Data Warehouse, Data Lake, and Huawei FusionInsight Intelligent Data Lake
Updated on 2023-03-01 GMT+08:00
View PDF

Database, Data Warehouse, Data Lake, and Huawei FusionInsight Intelligent Data Lake

As the Internet and IoT continue to evolve, data management tools are developing rapidly to cope with massive data. As a result, concepts related to big data are emerging, such as database, data warehouse, data lake, and lakehouse. This section describes these concepts and their relationships, as well as the corresponding Huawei products and solutions.

Database

A database is a warehouse where data is organized, stored, and managed based on the data structure.

In a broad sense, databases have been used in computers since the 1960s. However, hierarchical and network models were prevailing at that time, and they lacked structural independency between data and applications. As a result, database application was limited.

Nowadays, a database usually refers to a relational database. A relational database organizes data based on relational models and stores data in a set of tables with columns and rows. Therefore, data is well-structured and independent with low redundancy. In 1970, relational databases were born to completely separate data from applications in software and since then have become an indispensable part of mainstream computer systems. Relational databases have become one of the most important database products. Almost all new database products from database vendors support relational databases. Some non-relational database products also have APIs that support relational databases.

Relational databases process basic and routine transactions using OLTP, such as bank transactions.

Data Warehouse

The large-scale application of databases has facilitated the exponential growth of data. OLAP is desired more than ever to explore the relationship between data and unveil the untapped data value. However, it is difficult to share data between different databases, and data integration and analysis also face great challenges.

To overcome these challenges, Bill Inmon, the father of the data warehouse, proposed the idea of data warehousing in 1990. The data warehouse runs on a unique data storage architecture to perform OLAP on a large amount of data accumulated over the years. In this way, enterprises can obtain valuable information from massive data quickly and effectively to make informed decisions. The appearance of data warehouses has prompted the information industry to develop from operational systems based on relational databases to decision support systems.

Unlike a database, a data warehouse has the following features:

  • It is theme-oriented. It supports various services and operational data. Therefore, the required data needs to be extracted from multiple heterogeneous data sources, processed and integrated, and reorganized by the theme.
  • A data warehouse mainly supports enterprise decision analysis and operations involved are mainly data queries. Therefore, it improves the query speed and cuts down the total cost of ownership (TCO) by optimizing the table structures and storage modes.
Table 1 Comparison between data warehouses and databases

Dimension

Data Warehouse

Database

Application scenarios

OLAP

OLTP

Data source

Multiple

Single

Data normalization

Denormalized schemas

Highly normalized static schemas

Data access

Optimized read operations

Optimized write operations

Data Lake

Data is an important asset for enterprises. As production and operations data piles up, enterprises hope to save the data for effective management and centralized governance and explore data values.

The data lake provides a good answer to these requirements. It is a large data warehouse that centrally stores structured and unstructured data. It can store raw data from multiple sources and of multiple types. The data can be accessed, processed, analyzed, and transmitted without being structured. The data lake helps enterprises quickly complete federated analysis of heterogeneous data sources and explore data value.

A data lake is in essence a solution that consists of a data storage architecture and data processing tools.
  • The data storage architecture must be scalable and reliable enough to store massive structured, semi-structured, and unstructured data.
  • Data processing tools are classified into two types:
    • First type: focuses on how to migrate data into the lake, including defining data sources, formulating data synchronization policies, moving data, and compiling data catalogs.
    • Second type: focuses on how to analyze, explore, and utilize data in the lake. The data lake must be equipped with wide-ranging capabilities, such as comprehensive data and data lifecycle management, diversified data analytics, and secure data acquisition and release. Without these data governance tools, the quality of data in the lake cannot be guaranteed due to the lack of metadata. As a result, the data lake may turn into a data swamp.

With the development of big data and AI, the value of data in the data lake is increasing gradually. The data lake enables enterprises to build more optimized operation models by realizing centralized data management. It also helps enterprise with prediction analysis and recommendation models, stimulating further growth of enterprise capabilities.

The difference between a data warehouse and a data lake is analogous to that between a warehouse and a lake: A warehouse stores goods from a specific source while the water (raw data) in a lake comes from rivers, streams and other sources.

Table 2 Comparison between data lakes and data warehouses

Dimension

Data Lake

Data Warehouse

Application scenarios

Exploratory analytics, such as machine learning, data discovery, profiling, and prediction

Data analytics based on historical structured data

Cost

Low initial cost and high subsequent cost

High initial cost and low subsequent cost

Data quality

Massive raw data to be cleaned and normalized before use

High-quality data that can be used as the basis of facts

Target user

Data scientists and data developers

Business analysts

Huawei FusionInsight Intelligent Data Lake

Huawei's DAYU is a data enablement solution that helps large government agencies and enterprises customize their own intelligent data resource management solutions. This solution can import all-domain data into the data lake, eliminating data silos, unleashing the value of data, and empowering data-driven digital transformation.

DAYU, with the FusionInsight Intelligent Data Lake as the core, contains computing engines such as the database, data warehouse, and data lake, as well as platforms such as DataArts Studio. DAYU provides comprehensive data enablement capabilities, covering data collection, aggregation, computing, asset management, and data openness.

The FusionInsight Intelligent Data Lake solution provides the following services:

Parent topic: Consultation and Billing

Consultation and Billing FAQs

more