How to Perform Remote Debugging During MapReduce Secondary Development?

Question

During the secondary development of MapReduce, how to perform remote debugging?

Answer

MapReduce adopts Java remote debugging mechanism. Run Java remote debugging commands when starting the Map or Reduce tasks.

1. The mapreduce.map.java.opts and mapreduce.reduce.java.opts parameters specify the JVM startup parameters of Map and Reduce tasks respectively. In the client installation path/Yarn/config/mapred-site.xml configuration file on the client, add the command -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000 to the mapreduce.map.java.opts and mapreduce.reduce.java.opts parameters.
2. MapReduce is a distributed computing framework, and the node where Map or Reduce tasks are running are not fixed. It is advisable to keep only one NodeManager running in the cluster to ensure that the task is executed in the running NodeManager.
3. Submit MapReduce tasks on the client, then the Map or Reduce tasks will be suspended and listen to the port 8000 to wait for remote debugging.
4. In IDE, select the implementation class of MapReduce tasks and configure the remote debugging information to perform debugging.
   1. Double-click the area in the blue box to configure or cancel the break point.

      

   2. Right-click the break point and choose Debug As->Debug Configurations... from the shortcut menu.

      

   3. On the displayed page, double-click Remote Java Application and configure the Connection Properties area. Set Host to the IP address of the NodeManager and Port to 8000, and then click Debug.

      

If you use IDE to submit MapReduce tasks, the IDE is the client. Modify the mapred-site.xml file of the secondary development project by referring to 1.