Optimizing Job Parameters

Overview

The real-time data migration service uses the Flink stream processing framework and contains JobManager and TaskManager, which are the most important components of the Flink system.

If you adjust the parameters of a job, such as CPU Cores, Maximum Concurrent Requests, and Execution Memory, JobManager and TaskManager can be adjusted. By default, a job uses 2 CPUs and 8 GB memory, and a JobManager process and a TaskManager process are created, both of which use 1 CPU and 4 GB memory.

Job Optimization

The JobManager and TaskManager processes that both use 1 CPU and 4 GB memory can meet requirements in most scenarios. You can also modify the specifications of JobManager and TaskManager to meet requirements in special scenarios. A good case in point is the job memory overflow, where you can add custom attributes in the Configure Task area on the real-time migration job page and adjust the memory of JobManager and TaskManager to meet synchronization requirements.

Figure 1 Adding custom attributes
Click to enlarge

**Table 1** Job parameters
Parameter	Type	Default Value	Description
jobmanager.memory.process.size	int	3586 MB	Processing memory of JobManager, which directly affects the heap memory size NOTE: This memory occupies resources and may stop you from adding other jobs. Do not configure it unless necessary.
taskmanager.memory.process.size	int	3686 MB	Processing memory of TaskManager, which directly affects the heap memory size NOTE: This memory occupies resources and may stop you from adding other jobs. Do not configure it unless necessary.
taskmanager.memory.managed.fraction	int	0.2	Percentage of the TaskManager managed memory
taskmanager.memory.network.max	int	128 MB	This parameter is not required by default. If there are too many instances and tables in the database and table sharing scenario, you can increase the network memory as needed.
taskmanager.memory.network.fraction	int	0.1	This parameter is not required by default. If there are too many instances and tables in the database and table sharing scenario, you can increase the network memory as needed.
checkpoint.interval	int	60000	Interval at which a Flink job generates checkpoints, in milliseconds. For jobs with a large amount of data, you are advised to set this parameter to a larger value, which allows for a longer time for data flushing but increases the latency.
checkpoint.timeout.ms	int	600000	Timeout interval for a Flink job to generate checkpoints, in milliseconds