Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using Loader/ Common Loader Parameters
Updated on 2022-09-15 GMT+08:00
View PDF

Common Loader Parameters

This section applies to MRS 3.x or later.

Navigation Path

For details about the how to set parameters, see Modifying Cluster Service Configuration Parameters.

Parameter Description

Table 1 Common Loader parameters

Parameter

Description

Default Value

Value Range

mapreduce.client.submit.file.replication

Number of copies of the job files that the MapReduce task depends on in HDFS. If the number of DataNodes in the cluster is less than the value of this parameter, the number of copies is equal to the number of DataNodes. If the number of DataNodes is greater than or equal to the value of this parameter, the number of copies is the value of this parameter.

10

3 to 256

loader.fault.tolerance.rate

Error tolerance.

If the value is greater than 0, the error tolerance mechanism is enabled. When enabling the fault tolerance mechanism, you are advised to set the number of Map jobs to be greater than or equal to 3. It is recommended that this function be used when the job data volume is large.

0

0 to 1.0

loader.input.field.separator

Default input field separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used.

,

-

loader.input.line.separator

Default input line separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used.

-

-

loader.output.field.separator

Default output field separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used.

,

-

loader.output.line.separator

Line separator of data that Loader outputs

-

-
  • Because it needs time to calculate the fault tolerance rate, you are recommended to use the loader.fault.tolerance.rate parameter when the job runtime is longer than 2 minutes to ensure user experience.
  • Default separators are configured for the parameters in the preceding table for Loader. If separators are configured in the conversion steps for the jobs, the separators in the conversion steps will be used. If separators are not configured in the conversion steps, the default separators will be used.
Parent topic: Using Loader