Why Is an Error Reported When I Access the Native Page of an Application in Spark JobHistory?

Question

Submit a Spark application that contains millions of tasks in a single job. After the application is complete, if you access the native page of the application in JobHistory, the browser will wait for a long time before the native page of the application is displayed. If the native page cannot be displayed within 10 minutes, Proxy Error is displayed.

Figure 1 Example error information

Answer

When you switch to the native page of an application on the JobHistory page, JobHistory needs to replay the event log of the application. If the application contains a large number of event logs, the replay takes a long time and the browser needs to wait for a long time.

When the browser accesses the JobHistory native page, the httpd proxy is required. The proxy timeout interval is 10 minutes. Therefore, if JobHistory cannot parse the event log and return the event log within 10 minutes, httpd returns the Proxy Error message to the browser.

Solution

The local disk cache function is enabled for JobHistory. When an application is accessed, the event log parsing result of the application is cached to the local disk. When the application is accessed for the second time, the response speed is greatly accelerated. In this case, you only need to wait for a while and access the original link again. In this case, you do not need to wait for a long time.