Error Message "netty.exception.RemoteTransportException" Is Reported During Flink Task Running

Symptom

The error message "netty.exception.RemoteTransportException" is displayed during Flink job runtime.

Cause Analysis

• Possible cause 1:
  Full GC occurs at the heap memory level.

  1. Based on the error information in the logs, we can locate the node where the faulty TaskManager is.
  2. Check whether a large number of full GCs occur or the GC duration is too long in the GC logs of the node. As shown in the following figure, each GC takes more than 10s.

     

     Note: It is normal that Full GC occurs in the first several TaskManager startups. It is also normal that the full GC interval is long and the full GC duration is short. As shown in the following figure, the interval is at the hour level, and the interval does not exceed 1s.

     

• Possible cause 2:

  A thread-level memory leak (java.lang.OutOfMemoryError: unable to create new native thread) occurs on the service side.

  

  1. You can locate the node where the faulty TaskManager is deployed based on the logs of the abnormal job.
  2. The logs of the abnormal TaskManager node contain the error message unable to create new native thread type.
  3. This error indicates that the abnormal TaskManager node cannot create new threads. As a result, the TaskManager node on the current node is blocked.
• Possible cause 3:
  The hardware of NodeManager is faulty.

  1. You can locate the node where the faulty TaskManager is deployed based on the logs of the abnormal job.
  2. No exception information is found in the TaskManager log.
  3. A large number of disk exceptions are found in the OS logs of the NodeManager node where TaskManager is located.

     

  4. Check whether the yarn-start-stop.log file of the NodeManager process is restarted.

  

  In addition to the preceding error information, if the disk of the NodeManager node is full, the NodeManager process restarts and all containers fail. In this case, the Flink job also fails.

Procedure

• Solution to cause 1:
  1. Increase the startup memory of the TaskManager on the service side.
  2. If backpressure exists on the service side, increase the overall service concurrency to eliminate backpressure.

  If backpressure does not exist, check whether memory leakage occurs in the code on the service side. You are advised to enable the automatic restart. For example, set restart-strategy: failure-rate in the flink-conf.xml file.

• Solution to cause 2:
  1. Check whether the OS configuration of the node is improper.
  2. Determine whether memory leakage occurred based on the method of determining whether thread leakage occurred in Flink jobs. (Use jstack.)
• Solution to cause 3:
  1. Rectify the disk fault.
  2. You are advised to enable the automatic restart. For example, set restart-strategy: failure-rate in the flink-conf.xml file.