Scala

To avoid API compatibility or reliability issues after updates to the open-source Spark, it is advisable to use APIs of the version you are currently using.

Spark Core Common Interfaces

Spark mainly uses the following classes:

SparkContext: external interface of Spark, which is used to provide the functions of Spark for Scala applications that invoke this class, for example, connecting Spark clusters and generating RDDs.
SparkConf: Spark application configuration class, which is used to configure the application name, execution model, and executor memory.
Resilient Distributed Dataset (RDD): defines the RDD class in the Spark application. The class provides the data collection operation methods, such as map and filter.
PairRDDFunctions: provides computation operations for the RDD data of the key-value pair, such as groupByKey.
Broadcast: broadcast variable class. This class retains one read-only variable, and caches it on each machine, instead of saving a copy for each task.
StorageLevel: data storage level class, including MEMORY_ONLY, DISK_ONLY, and MEMORY_AND_DISK.

The RDD supports two types of operations, transformation and action. Table 1 and Table 2 show the common methods.

**Table 1** Transformation
Method	Description
map[U](f: (T) => U): RDD[U]	Return a new RDD by applying a function to all elements of this RDD.
filter(f: (T) => Boolean): RDD[T]	Return a new RDD containing only the elements that satisfy a predicate.
flatMap[U](f: (T) => TraversableOnce[U])(implicit arg0: ClassTag[U]): RDD[U]	Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
sample(withReplacement: Boolean, fraction: Double, seed: Long = Utils.random.nextLong): RDD[T]	Return a sampled subset of this RDD.
union(other: RDD[T]): RDD[T]	Return a new RDD, contains source RDD and the group of RDD's elements.
distinct([numPartitions: Int]): RDD[T]	Return a new RDD containing the distinct elements in this RDD.
groupByKey(): RDD[(K, Iterable[V])]	Group the values for each key in the RDD into a single sequence. Hash-partitions the resulting RDD with into numPartitions partitions.
reduceByKey(func: (V, V) => V[, numPartitions: Int]): RDD[(K, V)]	Merge the values for each key using an associative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce. Output will be hash-partitioned with numPartitions partitions.
sortByKey(ascending: Boolean = true, numPartitions: Int = self.partitions.length): RDD[(K, V)]	Sort the RDD by key, so that each partition contains a sorted range of the elements. Calling collect or save on the resulting RDD will return or output an ordered list of records (in the save case, they will be written to multiple part-X files in the filesystem, in order of the keys).
join[W](other: RDD[(K, W)][, numPartitions: Int]): RDD[(K, (V, W))]	Return an RDD containing all pairs of elements with matching keys in this and other. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this and (k, v2) is in other. Performs a hash join across the cluster.
cogroup[W](other: RDD[(K, W)], numPartitions: Int): RDD[(K, (Iterable[V], Iterable[W]))]	For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cartesian[U](other: RDD[U])(implicit arg0: ClassTag[U]): RDD[(T, U)]	Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.

**Table 2** Action
Method	Description
reduce(f: (T, T) => T):	Reduces the elements of this RDD using the specified commutative and associative binary operator.
collect(): Array[T]	Return an array that contains all of the elements in this RDD.
count(): Long	Return the number of elements in the RDD.
first(): T	Return the first element in this RDD.
take(num: Int): Array[T]	Take the first num elements of the RDD. This currently scans the partitions one by one, so it will be slow if a lot of partitions are required. In that case, use collect() to get the whole RDD instead.
takeSample(withReplacement: Boolean, num: Int, seed: Long = Utils.random.nextLong): Array[T]	Return a fixed-size sampled subset of this RDD in an array
saveAsTextFile(path: String): Unit	Save this RDD as a compressed text file, using string representations of elements.
saveAsSequenceFile(path: String, codec: Option[Class[_ <: CompressionCodec]] = None): Unit	Extra functions available on RDDs of (key, value) pairs to create a Hadoop SequenceFile, through an implicit conversion.
countByKey(): Map[K, Long]	Count the appearance times of each key.
foreach(func: (T) => Unit): Unit	Applies a function f to all elements of this RDD.
countByValue()(implicit ord: Ordering[T] = null): Map[T, Long]	Return the count of each unique value in this RDD as a map of (value, count) pairs. The final combine step happens locally on the master, equivalent to running a single reduce task.

**Table 3** New APIs of Spark core
API	Description
isSparkContextDown:AtomicBoolean	Determines whether sparkContext is shut down completely. The initial value is false. The value true indicates that sparkContext is shut down completely. The value false indicates that sparkContext is not shut down. For example, sc.isSparkContextDown.get() == true indicates that sparkContext is shut down completely.

Spark Streaming Common Interfaces

Spark Streaming mainly uses the following classes:

StreamingContext: main entrance of the Spark Streaming, which is used to provide methods for creating the DStream. Intervals by batch need to be set in the input parameter.
dstream.DStream: a type of data which indicates the RDDs continuous sequence. It indicates the continuous data flow.
dstream.PariDStreamFunctions: the Dstream of key-value, common operations are groupByKey and reduceByKey.
The cooperated Java APIs of Spark Streaming are JavaStreamingContext, JavaDStream, JavaPairDStream.

Common methods of Spark Streaming are the same as those of Spark Core. The following table describes some special Spark Streaming methods.

**Table 4** Spark Streaming methods
Method	Description
socketTextStream(hostname: String, port: Int, storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2): ReceiverInputDStream[String]	Reduces the elements of this RDD using the specified commutative and associative binary operator.
start():Unit	Return an array that contains all of the elements in this RDD.
awaitTermination(timeout: long):Unit	Return the number of elements in the RDD.
stop(stopSparkContext: Boolean, stopGracefully: Boolean): Unit	Return the first element in this RDD.
transform[T](dstreams: Seq[DStream[_]], transformFunc: (Seq[RDD[_]], Time) ? RDD[T])(implicit arg0: ClassTag[T]): DStream[T]	Take the first num elements of the RDD. This currently scans the partitions one by one, so it will be slow if a lot of partitions are required. In that case, use collect() to get the whole RDD instead.
UpdateStateByKey(func)	Return a fixed-size sampled subset of this RDD in an array
window(windowLength, slideInterval)	Save this RDD as a compressed text file, using string representations of elements.
countByWindow(windowLength, slideInterval)	Count the appearance times of each key.
reduceByWindow(func, windowLength, slideInterval)	Applies a function f to all elements of this RDD.
join(otherStream, [numTasks])	Return the count of each unique value in this RDD as a map of (value, count) pairs. The final combine step happens locally on the master, equivalent to running a single reduce task.
DStreamKafkaWriter.writeToKafka()	Writes data from the DStream into Kafka in batch.
DStreamKafkaWriter.writeToKafkaBySingle()	Writes data from the DStream into Kafka one by one.

**Table 5** Spark Streaming enhancement interface
Method	Description
DStreamKafkaWriter.writeToKafka()	Writes data from the DStream into Kafka in batch.
DStreamKafkaWriter.writeToKafkaBySingle()	Writes data from the DStream into Kafka one by one.

SparkSQL Common Interfaces

Spark SQL mainly uses the following classes:

SQLContext: main entrance of the Spark SQL function and DataFrame.
DataFrame: a distributed dataset organized by naming columns.
HiveContext: An instance of the Spark SQL execution engine that integrates with data stored in Hive.

**Table 6** Common Actions methods
Method	Description
collect(): Array[Row]	Return an array containing all DataFrame columns.
count(): Long	Return the number of DataFrame rows.
describe(cols: String*): DataFrame	Count the statistic information, including the counting, average value, standard deviation, minimum value and maximum value.
first(): Row	Return the first row.
Head(n:Int): Row	Return the first n rows.
show(numRows: Int, truncate: Boolean): Unit	Display the first 20 rows in table.
take(n:Int): Array[Row]	Return the first n rows in the DataFrame.

**Table 7** Basic DataFrame Functions
Method	Description
explain(): Unit	Print the logical plan and physical plan of the SQL.
printSchema(): Unit	Print the schema information to the console.
registerTempTable(tableName: String): Unit	Register the DataFrame as a temporary table, whose period is bound to the SQLContext.
toDF(colNames: String*): DataFrame	Return a DataFrame whose columns are renamed.