Help Center> Cloud Container Engine> User Guide> Old Console> Add-ons> spark-operator
View PDF

spark-operator

Introduction

Spark Operator is an operator who manages the lifecycle of Apache Spark applications on Kubernetes. It aims to specify and run Spark applications (workloads) as easily as running other types of workloads on Kubernetes.

Paired with Spark Operator, Kubernetes can better control and manage the lifecycle of Spark applications, including application status monitoring, log obtaining, and application running control. This allows the Spark on Kubernetes solution to be fully integrated with Kubernetes.

Spark Operator consists of the following components:

  1. SparkApplication controller: This controller is used to create, update, and delete SparkApplication objects. It also monitors events and performs actions accordingly.
  2. Submission Runner: This component invokes spark-submit to submit Spark jobs. The job submission process is the same as the Spark on Kubernetes solution.
  3. Spark Pod Monitor: This component monitors the status of pods related to Spark jobs and synchronizes the status to the controller.
  4. Muting Admission Webhook: (Optional) This component implements some customized requirements of the driver and executor pods based on annotations.
  5. SparkCtl: command line tool for interacting with the Spark Operator.

In addition to basic job submission, Spark Operator also supports the following features:

  • Declarative job management;
  • Automatic job re-submission after a SparkApplication object is updated;
  • Configurable restart policies;
  • Retry for a failed task submission;
  • Integration with Prometheus to collect and forward Spark application-level metrics and driver/executor pod metrics to Prometheus.

More information: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

Notes and Constraints

This add-on can be installed only in CCE clusters of v1.13.

Installing the Add-on

  1. Log in to the CCE console. In the navigation pane, choose Add-ons. On the Add-on Marketplace tab page, click Install Add-on under spark-operator.
  2. On the Install Add-on page, select the cluster and the add-on version, and click Next: Configuration.
  3. Click Install to directly install the add-on. Currently, the spark-operator add-on has no configurable parameters.

    After the add-on is installed, click Go Back to Previous Page. On the Add-on Instance tab page, select the corresponding cluster to view the running instance. This indicates that the add-on has been installed on each node in the cluster.

Upgrading the Add-on

  1. Log in to the CCE console. In the navigation pane, choose Add-ons. On the Add-on Instance tab page, click Upgrade under spark-operator.

    • If the Upgrade button is not available, the current add-on is already up-to-date and no upgrade is required.
    • During the upgrade, the spark-operator add-on of the original version on cluster nodes will be discarded, and the add-on of the target version will be installed.

  2. On the Basic Information page, select the add-on version and click Next.
  3. Click Upgrade.

Uninstalling the Add-on

  1. Log in to the CCE console. In the navigation pane, choose Add-ons. On the Add-on Instance tab page, click Uninstall under spark-operator.
  2. In the dialog box displayed, click Yes to uninstall the add-on.
Parent topic: Add-ons