Doris Application Development Overview

Common Concepts

In Doris, data is logically described in the form of tables.

• Row&Column

  A table consists of rows and columns:

  • Row: indicates a row of user data.
  • Column: describes different fields in a row of data.

    Columns can be divided into two categories: Key and Value. From the business perspective, Key and Value can correspond to the dimension column and indicator column respectively. From the point of view of the aggregation model, rows with the same key column are aggregated into one row. The aggregation mode of the Value column is specified by the user when creating a table.

• Tablet&Partition

  In the storage engine of Doris, user data is horizontally divided into several data fragments (tablets, also called data buckets). Each Tablet contains several rows of data. Tablet data has no intersection and is physically stored independently.

  Multiple tablets belong to different partitions logically. A tablet belongs to only one partition, and a partition contains several tablets. Because tablets are physically independent, partitions can be considered as physically independent. A tablet is the smallest physical storage unit for operations such as data movement and replication.

  Multiple partitions form a table. A partition can be considered as the smallest logical management unit. Data can be imported and deleted only for one partition.

• Data model

  Doris data models are classified into three types: Aggregate, Unique, and Duplicate.

  • Aggregate model

    When data is imported, rows with the same Key column are aggregated into one row, and the Value column is aggregated based on the AggregationType parameter. Currently, the AggregationType supports the following aggregation modes:

    • SUM: Sum of values in multiple lines.
    • Replace: Replace the value in the next batch of data.
    • MAX: retain the maximum value.
    • MIN: minimum value.
  • Unique model

    In some multidimensional analysis scenarios, users focus more on how to ensure the uniqueness of keys, that is, how to obtain the uniqueness constraint of the primary key. Therefore, the Unique data model is introduced.

    • Read-on-Merge

      The read-on-merge implementation of the Unique model can be completely replaced by the Replace mode in the Aggregate model. Its internal implementation and data storage are exactly the same.

    • Combine on write

      The unique model is implemented by combining data on write. The query performance of the unique model is closer to that of the duplicate model. Compared with the aggregate model, the unique model has higher query performance than the aggregate model in scenarios where primary key constraints are required, especially in aggregate queries and queries where a large amount of data needs to be filtered by indexes.

      In the Unique table with the merge-on-write option enabled, the overwritten and updated data is marked and deleted during the import phase, and new data is written to a new file. During query, all data marked for deletion is filtered out at the file level, and the read data is the latest data. This eliminates the data aggregation process during read merge and supports pushdown of multiple predicates in many cases. Therefore, the performance can be greatly improved in many scenarios, especially in the case of aggregated queries.

  • Duplicate model

    In some multidimensional analysis scenarios, data has neither primary keys nor aggregation requirements. The Duplicate data model can be introduced to meet such requirements.

    This data model is different from the Aggregate and Unique models. The data is stored exactly as the data in the import file, and there is no aggregation. Even if the two rows of data are identical, they are retained. The DUPLICATE KEY specified in the table creation statement is only used to specify the columns by which the underlying data is sorted.

  • Data Model Selection Suggestions

    The data model is determined when the table is created and cannot be modified. Therefore, it is important to choose a suitable data model.

    1. The Aggregate model greatly reduces the amount of data to be scanned and the amount of query calculation during aggregation query through pre-aggregation. It is suitable for the report query scenario with fixed mode. However, this model is not friendly to count(*) queries. In addition, the aggregation mode of the Value column is fixed. Therefore, the semantic correctness needs to be considered when performing other types of aggregation queries.
    2. The Unique model ensures the uniqueness of primary key constraints in scenarios where unique primary key constraints are required. However, the query advantages of pre-aggregation such as ROLLUP cannot be taken advantage of.
       1. For users who have high performance requirements for aggregated queries, the merge-on-write implementation introduced in version 1.2 is recommended.
       2. The Unique model supports only the entire row update. If the unique primary key constraint and some columns need to be updated, (e.g., importing multiple source tables into a Doris table), you can use the Aggregate model and set the aggregation type of the non-primary key column to Replaced_IF_NOT_NULL.
       3. Duplicate is applicable to Ad-hoc queries in any dimension. Although the pre-aggregation feature cannot be used, it is not restricted by the aggregation model and can take advantage of the column-store model. (Only the relevant columns are read, not all the Key columns).

Doris Sample Project Introduction

To obtain the MRS sample project, visit https://github.com/huaweicloud/huaweicloud-mrs-example. Switch to the version branch that matches the MRS cluster, download the package to the local PC, and decompress the package to obtain the sample code project of each component.