How to Perform Remote Debugging Using IDEA?

Question

How to perform remote debugging using IDEA during Spark secondary development?

Answer

The SparkPi application is used as an example here to illustrate how to perform remote debugging using IDEA.

1. Open the project and choose Run > Edit Configurations.
2. On the displayed window, click at the upper left corner. Then on the drop-down menu, choose Remote, as shown in Figure 1.
   Figure 1 Choosing Remote
   
3. Configure the Host, Port, and Search source using module's classpath, as shown in Figure 2.

   Host indicates the IP address of the Spark client and Port indicates the debugging port. Ensure that the port is available on the VM.

   Figure 2 Configuring parameters
   
   

   If the value of Port is changed, the debugging command of For JDK1.4.x must be changed accordingly. For example, if the value of Port is changed to 5006, the debugging command must be changed to -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006, which will be used during the startup of Spark.

4. Run the following command to remotely start SparkPi on the Spark client:

   ./spark-submit --master yarn-client --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006" --class org.apache.spark.examples.SparkPi /opt/FI-Client/Spark2x/spark/examples/jars/spark-examples_2.12-3.1.1-xxx.jar

   Change the --class and jar package in the preceding command to the --class and jar package of the actual application. Change the -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006 to the For JDK1.4.xdebugging command obtained from 3

   Figure 3 Command for running Spark
   
5. Set the debugging breakpoint.

   Click the blank area on the left of the code editing window to select the breakpoint of code. Figure 4 illustrates how to select the breakpoint of the code in row 29 of SparkPi.scala.

   Figure 4 Setting the breakpoint
   
6. Start the debugging.

   On the menu bar of IDEA, choose Run > Debug 'Unnamed' to open a debugging window. Start the debugging of SparkPi, for example, performing step-by-step debugging, checking call stack information, and tracking variable values, as shown in Figure 5.

   Figure 5 Debugging