Using the HetuEngine Cross-Domain Function

Using the HetuEngine Cross-Domain Function

HetuEngine provide unified standard SQL to implement efficient access to multiple data sources distributed in multiple regions (or data centers), shields data differences in the structure, storage, and region, and decouples data and applications.

Figure 1 HetuEngine cross-region functions

Key Technologies and Advantages

• No single-point failure bottleneck: HSFabric supports horizontal scale-out and multi-channel parallel transmission, maximizing the transmission rate. Cross-domain latency is no longer a bottleneck.
• Better computing resource utilization: Data compression and serialization tasks are delivered to Worker for parallel computing.
• Efficient serialization: The data serialization format is optimized to reduce the amount of data to be transmitted at the same data volume level.
• Streaming transmission: Based on HTTP 2.0 stream, ensure the universality of the HTTP protocol and reduce the repeated invoking of RPC during the transmission of a large amount of data.
• Resumable transmission: A large amount of data is prevented from being retransmitted after the connection is interrupted abnormally during data transmission.
• Traffic control: The network bandwidth occupied by data transmission can be limited by region to prevent other services from being affected due to exclusive traffic occupation in cross-region limited bandwidth scenarios.

HetuEngine Cross-Domain Function Usage

Prerequisites:

• At least one HSFabric instance has been deployed on the data nodes of the local and remote clusters.
• The nodes where the HSFabric instances of the local and remote clusters have been deployed can communicate with each other.

Procedure

1. Open the data source in the local domain. You can create a virtual schema to shield the real schema information and instance information of the physical data source in the local domain from remote access requests. The remote end can use the virtual schema name to access the data source in the local domain.

   CREATE VIRTUAL SCHEMA hive01.vschema01 WITH (
     catalog = 'hive01',
     schema = 'ins1'
   );

2. Register the data source of the HetuEngine type on the remote HetuEngine and add the local domain HetuEngine by referring to Configuring a HetuEngine Data Source.
3. Use cross-domain collaborative analysis.

   // 1. Open the hive1.ins2 data source on the remote HetuEngine.
   CREATE VIRTUAL SCHEMA hive1.vins2 WITH (
     catalog = 'hive1',
     schema = 'ins2'
   );
   
   // 2. Register three types of data sources, including Hive, GaussDB A, and HetuEngine, on HetuEngine in the local domain.
   hetuengine> show catalogs;
     Catalog  
   ----------  
   dws        
   hetuengine_dc          
   hive         
   hive_dg
   system       
   systemremote 
   (6 rows)
   
   // 3. Perform cross-source collaborative analysis on HetuEngine in the local domain.
   select * from hive_dg.schema1.table1 t1 join hetuengine_dc.vins2.table3 t2 join dws.schema02.table4 t3 on t1.name = t2.item and t2.id = t3.cardNo;