Spark on OBS Tuning

Updated on 2022-11-18 GMT+08:00

View PDF

In the scenario where a small number of requests are frequently sent from Spark on OBS to OBS, you can disable OBS monitoring to improve performance.

Modify the configuration in the core-site.xml file on the Spark client.

**Table 1** Parameter description
Parameter	Description	Default Value
fs.obs.metrics.switch	Specifies whether to report OBS monitoring metrics. true: enable false: disable	true
fs.obs.metrics.consumer	Specifies the processing mode of OBS monitoring metrics. org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider: indicates that OBS monitoring metrics are collected. org.apache.hadoop.fs.obs.DefaultMetricsConsumer: indicates that OBS monitoring metrics are not collected. To use the OBS monitoring function, ensure that the function of reporting OBS monitoring metrics is enabled.	org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider