Spark Scala APIs

Updated on 2024-10-23 GMT+08:00

View PDF

To avoid API compatibility or reliability issues after updates to the open-source Spark, it is advisable to use APIs of the version you are currently using. For details about APIs, see [conref:text]Target does not exist.

Spark Core Common Interfaces

Spark mainly uses the following classes:

SparkContext: external interface of Spark, which is used to provide the functions of Spark for Scala applications that invoke this class, for example, connecting Spark clusters and generating RDDs.
SparkConf: Spark application configuration class, which is used to configure the application name, execution model, and executor memory.
Resilient Distributed Dataset (RDD): defines the RDD class in the Spark application. The class provides the data collection operation methods, such as map and filter.
PairRDDFunctions: provides computation operations for the RDD data of the key-value pair, such as groupByKey.
Broadcast: broadcast variable class. This class retains one read-only variable, and caches it on each machine, instead of saving a copy for each task.
StorageLevel: data storage levels, including memory (MEMORY_ONLY), disk (DISK_ONLY), and memory+disk (MEMORY_AND_DISK).

RDD supports two types of operations, transformation and action. Table 1 and Table 2 describe the common methods.

**Table 1** Transformation
Method	Description
map[U](f: (T) => U): RDD[U]	Returns a new RDD by applying a function to all elements of this RDD.
filter(f: (T) => Boolean): RDD[T]	Invokes the f method for all RDD elements to generate a satisfied data set that is returned in the form of RDD.
flatMap[U](f: (T) => TraversableOnce[U])(implicit arg0: ClassTag[U]): RDD[U]	Returns a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
sample(withReplacement: Boolean, fraction: Double, seed: Long = Utils.random.nextLong): RDD[T]	Returns a sampled subset of this RDD.
union(other: RDD[T]): RDD[T]	Returns a new RDD, contains source RDD and the group of RDD's elements.
distinct([numPartitions: Int]): RDD[T]	Returns a new RDD containing the distinct elements in this RDD.
groupByKey(): RDD[(K, Iterable[V])]	Returns (K,Iterable[V]) and combines the values of the same key to a set.
reduceByKey(func: (V, V) => V[, numPartitions: Int]): RDD[(K, V)]	Invokes func on the values of the same key.
sortByKey(ascending: Boolean = true, numPartitions: Int = self.partitions.length): RDD[(K, V)]	Sorts by key in ascending or descending order. Ascending is of the boolean type.
join[W](other: RDD[(K, W)][, numPartitions: Int]): RDD[(K, (V, W))]	Returns the dataset of (K,(V,W)) when the (K,V) and (K,W) datasets exist. numPartitions indicates the number of concurrent tasks.
cogroup[W](other: RDD[(K, W)], numPartitions: Int): RDD[(K, (Iterable[V], Iterable[W]))]	Returns the dataset of (K, (Iterable[V], Iterable[W])) when the (K,V) and (K,W) datasets of two key-value pairs exist. numPartitions indicates the number of concurrent tasks.
cartesian[U](other: RDD[U])(implicit arg0: ClassTag[U]): RDD[(T, U)]	Returns the Cartesian product of the RDD and other RDDs.

**Table 2** Action
API	Description
reduce(f: (T, T) => T):	Invokes f on elements of the RDD.
collect(): Array[T]	Returns an array that contains all of the elements in this RDD.
count(): Long	Returns the number of elements in the dataset.
first(): T	Returns the first element in this RDD.
take(num: Int): Array[T]	Returns the first n elements in this RDD.
takeSample(withReplacement: Boolean, num: Int, seed: Long = Utils.random.nextLong): Array[T]	Samples the dataset randomly and returns a dataset of num elements. withReplacement indicates whether replacement is used.
saveAsTextFile(path: String): Unit	Writes the dataset to a text file, HDFS, or file system supported by HDFS. Spark converts each record to a row of records and then writes it to the file.
saveAsSequenceFile(path: String, codec: Option[Class[_ <: CompressionCodec]] = None): Unit	This API can be used only on the key-value pair, and then it generates SequenceFile and writes the file to the local or Hadoop file system.
countByKey(): Map[K, Long]	Counts the appearance times of each key.
foreach(func: (T) => Unit): Unit	Applies a function f to all elements of this RDD.
countByValue()(implicit ord: Ordering[T] = null): Map[T, Long]	Counts the times that each element of the RDD occurs.

**Table 3** New APIs of Spark core
API	Description
isSparkContextDown:AtomicBoolean	Determines whether sparkContext is shut down completely. The initial value is false. The value true indicates that sparkContext is shut down completely. The value false indicates that sparkContext is not shut down. For example, sc.isSparkContextDown.get() == true indicates that sparkContext is shut down completely.

Spark Streaming Common Interfaces

Spark Streaming mainly uses the following classes:

StreamingContext: main entrance of the Spark Streaming, which is used to provide methods for creating the DStream. Intervals by batch need to be set in the input parameter.
dstream.DStream: a type of data which indicates the RDDs continuous sequence. It indicates the continuous data flow.
dstream.PariDStreamFunctions: the Dstream of key-value, common operations are groupByKey and reduceByKey.
The cooperated Java APIs of Spark Streaming are JavaStreamingContext, JavaDStream, JavaPairDStream.

Common methods of Spark Streaming are the same as those of Spark Core. The following table describes some special Spark Streaming methods.

**Table 4** Spark Streaming methods
Method	Description
socketTextStream(hostname: String, port: Int, storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2): ReceiverInputDStream[String]	Creates an input stream from the TCP source host:port.
start():Unit	Starts Spark Streaming computing.
awaitTermination(timeout: long):Unit	Terminates the await of the process, which is similar to pressing Ctrl+C.
stop(stopSparkContext: Boolean, stopGracefully: Boolean): Unit	Stops the Spark Streaming computing.
transform[T](dstreams: Seq[DStream[_]], transformFunc: (Seq[RDD[_]], Time) ? RDD[T])(implicit arg0: ClassTag[T]): DStream[T]	Performs the function operation on each RDD to obtain a new DStream.
UpdateStateByKey(func)	Updates the status of DStream. To use this method, you need to define the state and state update functions.
window(windowLength, slideInterval)	Generates a new DStream by batch calculating according to the window of the source DStream.
countByWindow(windowLength, slideInterval)	Returns the number of sliding window elements in the stream.
reduceByWindow(func, windowLength, slideInterval)	When the key-value pair of DStream is invoked, a new key-value pair of DStream is returned. The value of each key is obtained by aggregating the reduce function in batches in the sliding window.
join(otherStream, [numTasks])	Performs a join operation between different Spark Streamings.
DStreamKafkaWriter.writeToKafka()	Writes data from the DStream into Kafka in batch.
DStreamKafkaWriter.writeToKafkaBySingle()	Writes data from the DStream into Kafka one by one.

**Table 5** Spark Streaming enhancement interface
Method	Description
DStreamKafkaWriter.writeToKafka()	Writes data from the DStream into Kafka in batch.
DStreamKafkaWriter.writeToKafkaBySingle()	Writes data from the DStream into Kafka one by one.

SparkSQL Common Interfaces

Spark SQL mainly uses the following classes:

SQLContext: main entrance of the Spark SQL function and DataFrame.
DataFrame: a distributed dataset organized by naming columns.
HiveContext: main entrance for obtaining data stored in Hive.

**Table 6** Common Actions methods
Method	Description
collect(): Array[Row]	Returns an array containing all DataFrame columns.
count(): Long	Returns the number of DataFrame rows.
describe(cols: String*): DataFrame	Counts the statistic information, including the counting, average value, standard deviation, minimum value and maximum value.
first(): Row	Returns the first row.
Head(n:Int): Row	Returns the first n rows.
show(numRows: Int, truncate: Boolean): Unit	Displays DataFrame in a table.
take(n:Int): Array[Row]	Returns the first n rows in the DataFrame.

**Table 7** Basic DataFrame Functions
Method	Description
explain(): Unit	Prints the logical plan and physical plan of the SQL.
printSchema(): Unit	Prints the schema information to the console.
registerTempTable(tableName: String): Unit	Registers the DataFrame as a temporary table, whose period is bound to the SQLContext.
toDF(colNames: String*): DataFrame	Returns a DataFrame whose columns are renamed.

Parent topic: Common Spark APIs

Previous topic: Spark Java APIs

Next topic: Spark Python APIs

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot