Updated on 2024-08-30 GMT+08:00

Typical Flink Parameters

Table 1 Typical Flink parameters

Parameter

Description

Recommended Value

Description

-c

Main class name

Set this parameter as needed.

Yes

-yjm

Memory of the JobManager process. The default value is 2 GB.

Set this parameter as needed.

No

-ytm

Memory of the TaskManager process. The default value is 4 GB.

Set this parameter as needed.

No

-ynm

Name of the Flink YARN job.

Set this parameter as needed.

Yes

-ys

Number of slots in the TaskManager.

2

No

execution.checkpointing.interval

Interval for triggering a checkpoint, which can be added using -yD. The unit is ms.

60000

Yes

execution.checkpointing.timeout

Checkpoint timeout interval. You can run the -yD command to add a checkpoint timeout interval. The default value is 30 minutes.

30min

Yes

execution.checkpointing.tolerable-failed-checkpoints

Total number of tolerable checkpoint failures, which can be added using -yD.

1000

No

state.checkpoints.num-retained

Number of reserved checkpoints, which can be added using -yD.

5

No

state.backend

RocksDB used as the state backend, which can be added using -yD.

rocksdb

Enabled

state.backend.incremental

Incremental RocksDB state backends, which can be added using -yD.

TRUE

Yes

state.backend.rocksdb.block.blocksize

Size of data blocks on the write state backend. The value can be added using -yD.

512KB

Yes

state.backend.rocksdb.block.cache-size

Size of the block cache on the state backend, which can be added using -yD.

1024MB

Yes

taskmanager.memory.jvm-overhead.max

Maximum size of the local memory used for other JVM overheads, such as stack space and garbage collection space. The value can be added using -yD.

10g

No

taskmanager.memory.jvm-overhead.fraction

Ratio of the local memory used for other JVM overheads to TM memory, such as stack space and garbage collection space. The value can be added using -yD.

0.2

No

parallelism.default

Job parallelism. For example, to add the job parallelism for the join operator, use -yD. The default value is 1.

Set this parameter as needed.

No

table.exec.state.ttl

TTL (join ttl) of Flink state, which can be added using -yD. The default value is 0.

Set this parameter as needed.

Yes

heartbeat.timeout

Heartbeat timeout interval between the JM and TM, which can be added using -yD.

1800000

Yes

akka.ask.timeout

Akka communication timeout, which can be added using -yD.

240s

Yes

taskmanager.memory.segment-size

Size (in bytes) of the memory buffer block used by the memory management and network stack. The default value is 32768 (32 KB). You can add the value using -yD.

64kb

No

taskmanager.network.memory.max-buffers-per-channel

Maximum number of buffers that each channel can hold. If there are many idle segments, you can set a larger value. Otherwise, the channel will be blocked because it cannot obtain segments. You can run -yD to set this parameter.

100

No

taskmanager.network.memory.buffers-per-channel

Number of buffers exclusively used by each channel, which can be added by using -yD.

10

No

taskmanager.network.memory.floating-buffers-per-gate

Number of floating buffers used by each channel, which can be added by using -yD.

2000

No

taskmanager.network.netty.server.numThreads

Number of Netty server threads in each TaskManager, which can be added using -yD.

20

No

taskmanager.network.netty.client.numThreads

Number of Netty client threads in each TaskManager, which can be added using -yD.

20

No

state.backend.rocksdb.files.open

Maximum number of files that can be opened. The value -1 indicates that there is no limit. You can run -yD to set this parameter.

-1

No

state.backend.rocksdb.compaction.level.use-dynamic-size

Dynamic data volume threshold of data storage at each layer. You can use -yD to set this parameter.

TRUE

No

state.backend.rocksdb.levels.num

Number of compaction layers allowed by RocksDB, which can be added using -yD.

10

No

state.backend.rocksdb.compaction.style

Compaction algorithm, which can be added using -yD.

FIFO

No

state.backend.rocksdb.verify.checksum

Whether to disable data check during read. You can run -yD to set this parameter.

FALSE

No

state.backend.rocksdb.thread.num

Maximum number of concurrent threads responsible for flushing and compaction in the background. You can run -yD to set this parameter.

4

No

state.backend.rocksdb.writebuffer.count

Maximum number of memtables, which can be added using -yD.

5

No

state.backend.rocksdb.writebuffer.number-to-merge

Minimum number of memtables that are merged before flush. The value can be added using -yD.

3

No

state.backend.rocksdb.background.compaction.max

Maximum number of compaction threads, which can be added using -yD.

10

No

state.backend.rocksdb.flush.max

Number of RocksDB flush threads, which can be added using -yD.

1

No