Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ Spark Development Guide/ FAQs About Spark Application Development/ How Do I Perform Remote Debugging Using IDEA?
Updated on 2024-08-16 GMT+08:00
View PDF
Share

How Do I Perform Remote Debugging Using IDEA?

Question

How do I perform remote debugging using IDEA during Spark secondary development?

Answer

The SparkPi application is used as an example to illustrate how to perform remote debugging using IDEA.

  1. Open the project and choose Run > Edit Configurations.
  2. In the displayed window, click + at the upper left corner. Then on the drop-down menu, choose Remote, as shown in Figure 1.
    Figure 1 Choosing remote
  3. Configure the Host, Port, and search source using module's classpath, as shown in Figure 2.

    Host indicates the IP address of the Spark client and Port indicates the debugging port. Ensure that the port is available on the VM.

    Figure 2 Configuring parameters

    If the value of Port is changed, the debugging command of For JDK1.4.x must be changed accordingly. For example, if the value of Port is changed to 5006, the debugging command must be changed to -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006, which will be used during the startup of Spark.

  4. Run the following command to remotely start SparkPi on the Spark client:

    ./spark-submit --master yarn-client --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006" --class org.apache.spark.examples.SparkPi /opt/client/Spark/spark/examples/jars/spark-examples-<version>.jar

    • org.apache.spark.examples.SparkPi,opt/client/Spark/spark/examples/jars/spark-examples-<version>.jar: You need to change it to your own main class and JAR file path.
    • -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5006: Change the port to the commissioning port corresponding to For JDK1.4.x obtained in 3.
  5. Set the debugging breakpoint.

    Click the blank area on the left of the IDEA code editing window to select the breakpoint of code. Figure 4 illustrates how to select the breakpoint of the code in row 29 of SparkPi.scala.

    Figure 3 Setting the Breakpoint
  6. Start the debugging.

    On the menu bar of IDEA, choose Run > Debug 'Unnamed' to open a debugging window. Start the debugging of SparkPi, for example, performing step-by-step debugging, checking call stack information, and tracking variable values, as shown in Figure 5.

    Figure 4 Debugging