What Should I Do If the Data Volume of a Broker Load Import Task Exceeds the Threshold?

Symptom

The following error message was displayed when Broker Load was used to import data:

Scan bytes per broker scanner exceed limit:xxx

Cause Analysis

The maximum data volume of a single import task processed by the BE process is 3 GB. You can adjust the import parameters of Broker Load for large files.

Procedure

Modify the maximum scan amount and maximum concurrent number of tasks of a single BE node according to the current number of BE instances and the size of the files to be imported. Perform the following steps to modify the parameters:

1. Log in to FusionInsight Manager and choose Cluster > Services > Doris. On the dashboard page, view the IP address of leader host to determine the node where the active FE node is deployed.
2. Click Instance, and click the BE instance whose IP address was viewed in 1. Click Configurations > All Configurations, select BE (Role) > Customization, and add the following parameters:
   • max_broker_concurrency: number of BE nodes
   • max_bytes_per_broker_scanner: size of the file to be imported/number of BE nodes
   

   The configuration items take effect only when they are modified in the fe.conf file of the leader FE.

3. Click Save to save the configuration and restart the instance whose configuration has expired.