Spark on OBS Performance Tuning

Scenarios

In the scenario where a small number of requests are frequently sent from Spark on OBS to OBS, you can disable OBS monitoring to improve performance.

Configuration

1. Install the Spark client.
2. Log in to the Spark client node as the client installation user.
   Modify the following parameters in the Client installation directory/Spark/spark/conf/core-site.xml file on the Spark client.

   Table 1 Parameter description

   Parameter	Description	Default Value
   fs.obs.metrics.switch	Specifies whether to report OBS monitoring metrics. true: enable false: disable	true
   fs.obs.metrics.consumer	Specifies the processing mode of OBS monitoring metrics. org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider: indicates that OBS monitoring metrics are collected. org.apache.hadoop.fs.obs.DefaultMetricsConsumer: indicates that OBS monitoring metrics are not collected. To use the OBS monitoring function, ensure that the function of reporting OBS monitoring metrics is enabled.	org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider