Spark Scala APIs

Updated on 2024-08-16 GMT+08:00

View PDF

To avoid API compatibility or reliability problems, you are advised to use open source APIs of the corresponding version.

Common Spark Core APIs

Spark mainly uses the following classes:

SparkContext: external interface of Spark, which is used to provide the functions of Spark for Scala applications that invoke this class, for example, connecting Spark clusters and creating RDDs.
SparkConf: Spark application configuration class, which is used to configure the application name, execution model, and executor memory.
RDD: defines the RDD class in the Spark application. The class provides the data collection operation methods, such as map and filter.
PairRDDFunctions: provides computation operations for the RDD data of the key-value pair, such as groupByKey.
Broadcast: broadcast variable class. This class retains one read-only variable, and caches it on each machine, instead of saving a copy for each task.
StorageLevel: data storage levels, including memory (MEMORY_ONLY), disk (DISK_ONLY), and memory+disk (MEMORY_AND_DISK)

RDD supports two types of operations: Transformation and Action. Table 1 and Table 2 describe their common methods.

**Table 1** Transformation
Method	Description
map[U](f: (T) => U): RDD[U]	Uses the f method to generate a new RDD for each element in the RDD that invokes map.
filter(f: (T) => Boolean): RDD[T]	Invokes the f method for all RDD elements to generate a satisfied data set that is returned in the form of RDD.
flatMap[U](f: (T) => TraversableOnce[U])(implicit arg0: ClassTag[U]): RDD[U]	Invokes the f method for all RDD elements and then flattens the results to generate a new RDD.
sample(withReplacement: Boolean, fraction: Double, seed: Long = Utils.random.nextLong): RDD[T]	Samples and returns a subset of RDD.
union(other: RDD[T]): RDD[T]	Returns a new RDD that contains a set of elements of the source RDD and the specified RDD.
distinct([numPartitions: Int]): RDD[T]	Deletes duplicate elements to generate a new RDD.
groupByKey(): RDD[(K, Iterable[V])]	Returns (K,Iterable[V]) and combines the values of the same key to a set.
reduceByKey(func: (V, V) => V[, numPartitions: Int]): RDD[(K, V)]	Invokes func on the values of the same key.
sortByKey(ascending: Boolean = true, numPartitions: Int = self.partitions.length): RDD[(K, V)]	Sorts by key in ascending or descending order. Ascending is of the boolean type.
join[W](other: RDD[(K, W)][, numPartitions: Int]): RDD[(K, (V, W))]	Returns the dataset of (K,(V,W)) when the (K,V) and (K,W) datasets exist. numPartitions indicates the number of concurrent tasks.
cogroup[W](other: RDD[(K, W)], numPartitions: Int): RDD[(K, (Iterable[V], Iterable[W]))]	Returns the dataset of (K, (Iterable[V], Iterable[W])) when the (K,V) and (K,W) datasets of two key-value pairs exist. numPartitions indicates the number of concurrent tasks.
cartesian[U](other: RDD[U])(implicit arg0: ClassTag[U]): RDD[(T, U)]	Returns the Cartesian product of the RDD and other RDDs.

**Table 2** Action
Method	Description
reduce(f: (T, T) => T):	Invokes f on elements of the RDD.
collect(): Array[T]	Returns an array that contains all elements of the RDD.
count(): Long	Returns the number of elements in the dataset.
first(): T	Returns the first element in the dataset.
take(num: Int): Array[T]	Returns the first N elements.
takeSample(withReplacement: Boolean, num: Int, seed: Long = Utils.random.nextLong): Array[T]	Samples the dataset randomly and returns a dataset of num elements. withReplacement indicates whether replacement is used.
saveAsTextFile(path: String): Unit	Writes the dataset to a text file, HDFS, or file system supported by HDFS. Spark converts each record to a row of records and then writes it to the file.
saveAsSequenceFile(path: String, codec: Option[Class[_ <: CompressionCodec]] = None): Unit	This can be used only on the key-value pair, and then generates SequenceFile and writes is to the local or Hadoop file system.
countByKey(): Map[K, Long]	Counts the times that each key occurs.
foreach(func: (T) => Unit): Unit	Runs func on each element of the dataset.
countByValue()(implicit ord: Ordering[T] = null): Map[T, Long]	Counts the times that each element of the RDD occurs.

Common Spark Streaming APIs

Spark Streaming mainly uses the following classes:

StreamingContext: main entrance of Spark Streaming. It provides methods for creating the DStream. A batch interval needs to be set in the input parameter.
dstream.DStream: a type of data which indicates the RDDs continuous sequence. It indicates the continuous data flow.
dstream.PariDStreamFunctions: DStream of the key-value pair. Common operations include groupByKey and reduceByKey.
The Java APIs of the Spark Streaming are JavaSteamingContext, JavaDStream, and JavaPairDStream.

The common methods of Spark Streaming are similar to those of Spark Core. The following table provides some methods of Spark Streaming.

**Table 3** Spark Streaming methods
Method	Description
socketTextStream(hostname: String, port: Int, storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2): ReceiverInputDStream[String]	Creates an input stream using the TCP protocol (source host:port).
start():Unit	Starts the Streaming calculation.
awaitTermination(timeout: long):Unit	Terminates the await of the process, which is similar to pressing Ctrl+C.
stop(stopSparkContext: Boolean, stopGracefully: Boolean): Unit	Stops the Streaming calculation.
transform[T](dstreams: Seq[DStream[_]], transformFunc: (Seq[RDD[_]], Time) ? RDD[T])(implicit arg0: ClassTag[T]): DStream[T]	Performs the function operation on each RDD to obtain a new DStream.
UpdateStateByKey(func)	Updates the status of DStream. To use this method, you need to define the state and state update functions.
window(windowLength, slideInterval)	Generates a new DStream by batch calculating according to the window of the source DStream.
countByWindow(windowLength, slideInterval)	Returns the number of sliding window elements in the stream.
reduceByWindow(func, windowLength, slideInterval)	When the key-value pair of DStream is invoked, a new key-value pair of DStream is returned. The value of each key is obtained by aggregating the reduce function in batches in the sliding window.
join(otherStream, [numTasks])	Performs a join operation between different Spark Streamings.
DStreamKafkaWriter.writeToKafka()	Writes data from DStream into Kafka in batch.
DStreamKafkaWriter.writeToKafkaBySingle()	Writes data from DStream into Kafka one by one.

**Table 4** Streaming enhanced feature APIs
Method	Description
DStreamKafkaWriter.writeToKafka()	Writes data from DStream into Kafka in batch.
DStreamKafkaWriter.writeToKafkaBySingle()	Writes data from DStream into Kafka one by one.

Common Spark SQL APIs

Spark SQL mainly uses the following classes:

SQLContext: main entrance of Spark SQL functions and DataFrame
DataFrame: distributed dataset organized by naming columns
HiveContext: main entrance for obtaining data stored in Hive.

**Table 5** Common Actions methods
Method	Description
collect(): Array[Row]	Returns an array containing all the columns of DataFrame.
count(): Long	Returns the number of rows in DataFrame.
describe(cols: String*): DataFrame	Calculates the statistics, including the count, average value, standard deviation, minimum value, and maximum value.
first(): Row	Returns the first row.
Head(n:Int): Row	Returns the first n rows.
show(numRows: Int, truncate: Boolean): Unit	Displays DataFrame in a table.
take(n:Int): Array[Row]	Returns the first n rows in DataFrame.

**Table 6** Basic DataFrame functions
Method	Description
explain(): Unit	Prints the logical plan and physical plan of the SQL statement.
printSchema(): Unit	Prints schema information to the console.
registerTempTable(tableName: String): Unit	Registers DataFrame as a temporary table, whose period is bound to SQLContext.
toDF(colNames: String*): DataFrame	Returns a DataFrame whose columns are renamed.