Updated on 2025-08-22 GMT+08:00

Spark on OBS Performance Tuning

Scenarios

In the scenario where a small number of requests are frequently sent from Spark on OBS to OBS, you can disable OBS monitoring to improve performance.

Configuration

  1. Install the Spark client.

    For details, see Installing a Client.

  2. Log in to the Spark client node as the client installation user.

    Modify the following parameters in the Client installation directory/Spark/spark/conf/core-site.xml file on the Spark client.
    Table 1 Parameter description

    Parameter

    Description

    Default Value

    fs.obs.metrics.switch

    Specifies whether to report OBS monitoring metrics.

    • true: enable
    • false: disable

    true

    fs.obs.metrics.consumer

    Specifies the processing mode of OBS monitoring metrics.

    • org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider: indicates that OBS monitoring metrics are collected.
    • org.apache.hadoop.fs.obs.DefaultMetricsConsumer: indicates that OBS monitoring metrics are not collected.

    To use the OBS monitoring function, ensure that the function of reporting OBS monitoring metrics is enabled.

    org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider