Help Center> Cloud Container Engine> User Guide> New Console> Add-ons> spark-operator
View PDF

spark-operator

Introduction

Spark Operator is an operator who manages the lifecycle of Apache Spark applications on Kubernetes. It aims to specify and run Spark applications (workloads) as easily as running other types of workloads on Kubernetes.

Paired with Spark Operator, Kubernetes can better control and manage the lifecycle of Spark applications, including application status monitoring, log obtaining, and application running control. This allows the Spark on Kubernetes solution to be fully integrated with Kubernetes.

Spark Operator consists of the following components:

  1. SparkApplication controller: This controller is used to create, update, and delete SparkApplication objects. It also monitors events and performs actions accordingly.
  2. Submission Runner: This component invokes spark-submit to submit Spark jobs. The job submission process is the same as the Spark on Kubernetes solution.
  3. Spark Pod Monitor: This component monitors the status of pods related to Spark jobs and synchronizes the status to the controller.
  4. Muting Admission Webhook: (Optional) This component implements some customized requirements of the driver and executor pods based on annotations.
  5. SparkCtl: command line tool for interacting with the Spark Operator.

In addition to basic job submission, Spark Operator also supports the following features:

  • Declarative job management;
  • Automatic job re-submission after a SparkApplication object is updated;
  • Configurable restart policies;
  • Retry for a failed task submission;
  • Integration with Prometheus to collect and forward Spark application-level metrics and driver/executor pod metrics to Prometheus.

More information: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

Notes and Constraints

This add-on can be installed only in CCE clusters of v1.13.

Installing the Add-on

  1. Log in to the CCE console and access the cluster details page. Choose Add-ons in the navigation pane, locate spark-operator on the right, and click Install.
  2. Click Install to directly install the add-on. Currently, the spark-operator add-on has no configurable parameters.
Parent topic: Add-ons