Using the HetuEngine Cross-Source Function

Using the HetuEngine Cross-Source Function

Enterprises usually store massive data, such as from various databases and warehouses, for management and information collection. However, diversified data sources, hybrid dataset structures, and scattered data storage rise the development cost for cross-source query and prolong the cross-source query duration.

HetuEngine provides unified standard SQL statements to implement cross-source collaborative analysis, simplifying cross-source analysis operations.

Figure 1 HetuEngine cross-source function

Key Technologies and Advantages

• Compute pushdown: When HetuEngine is used for cross-source collaborative analysis, HetuEngine enhances the compute pushdown capability from the dimensions listed in the following table to improve access efficiency.
  • Basic pushdown: predicate, projection, subquery, and limit
  • Aggregate pushdown: GROUP BY, ORDER BY, COUNT, SUM, MIN, and MAX
  • Operator pushdown: <, >, LIKE, and OR.

Usage Guide of Cross-Source Function

HetuEngine supports quick joint query of multiple data sources and GUI-based data source configuration and management. You can quickly add the following data sources on the HSConsole page by referring to Before You Start:

• Configuring a Hive Data Source
• Configuring a Hudi Data Source
• Configuring a ClickHouse Data Source
• Configuring an Elasticsearch Data Source
• Configuring a GaussDB Data Source
• Configuring an HBase Data Source
• Configuring a HetuEngine Data Source
• Configuring an IoTDB Data Source
• Configuring a MySQL Data Source

Process of Using Cross-Source Collaborative Analysis

1. Log in to the HetuEngine client by referring to Using the HetuEngine Client.
2. Register data sources such as Hive, HBase, and GaussDB A.

   hetuengine> show catalogs;
     Catalog  
   ----------  
   dws                
   hive         
   hive_dg
   hbase
   system       
   systemremote 
   (6 rows)

3. Compile SQL statements for cross-source collaborative analysis.

   select * from hive_dg.schema1.table1 t1 join hbase.schema3.table3 t2 join dws.schema02.table4 t3 on t1.name = t2.item and t2.id = t3.cardNo;