How to Perform Remote Debugging Using IDEA?

Question

How to perform remote debugging using IDEA during Spark secondary development?

Answer

The SparkPi application is used as an example here to illustrate how to perform remote debugging using IDEA.

1. Open the project and choose Run > Edit Configurations.
2. In the displayed window, click at the upper left corner, and then choose Remote on the drop-down menu, as shown in Figure 1.
   Figure 1 Choosing Remote
   
3. Configure the Host, Port, and Search source using module's classpath, as shown in Figure 2.

   Host indicates the IP address of the Spark client and Port indicates the debugging port. Ensure that the port is available on the VM.

   Figure 2 Configuring parameters
   
   

   If the value of Port is changed, the debugging command of For JDK1.4.x must be changed accordingly. For example, if the value of Port is changed to 5006, the debugging command must be changed to -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006, which will be used during the startup of Spark.

4. Run the following command to remotely start SparkPi on the Spark client:

   ./spark-submit --master yarn-client --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006" --class org.apache.spark.examples.SparkPi /opt/FI-Client/Spark2x/spark/examples/jars/spark-examples_2.12-3.1.1-xxx.jar

   Change the --class and JAR package in the preceding command to the --class and JAR package of the actual application. Change the -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006 to the For JDK1.4.x debugging command obtained in 3.

   Figure 3 Command for running Spark
   
5. Set the debugging breakpoint.

   Click the blank area on the left of the code editing window to select the breakpoint of code. Figure 4 illustrates how to select the breakpoint of the code in row 29 of SparkPi.scala.

   Figure 4 Setting the breakpoint
   
6. Start the debugging.

   On the menu bar of IDEA, choose Run > Debug 'Unnamed' to open a debugging window. Start the debugging of SparkPi, for example, performing step-by-step debugging, checking call stack information, and tracking variable values, as shown in Figure 5.

   Figure 5 Debugging