Python

To avoid API compatibility or reliability issues after updates to the open-source Spark, it is advisable to use APIs of the version you are currently using.

Spark Core Common Interfaces

Spark mainly uses the following classes:

pyspark.SparkContext: external interface of Spark, which is used to provide the functions of Spark for Java applications that invoke this class, for example, connecting Spark clusters and generating RDDs, accumulations, and broadcasts. It functions as a container.
pyspark.SparkConf: Spark application configuration class, which is used to configure the application name, execution model, and executor memory.
pyspark.RDD: class used to define the RDD in the Spark application. The class provides the data collection operation methods, such as map and filter.
pyspark.Broadcast: A broadcast variable created with SparkContext.broadcast(). Access its value through value.
pyspark.StorageLevel: data storage level class, including MEMORY_ONLY, DISK_ONLY, and MEMORY_AND_DISK.
pyspark.sql.SQLContext: Main entry point for SparkSQL functionality. A SQLContext can be used create SchemaRDDs, register SchemaRDDs as tables, execute SQL over tables, cache tables, and read parquet files.
pyspark.sql.DataFrame: A distributed collection of data grouped into named columns. A DataFrame is equivalent to a relational table in Spark SQL, and can be created using various functions in SQLContext.
pyspark.sql.DataFrameNaFunctions: Functionality for working with missing data in DataFrame.
pyspark.sql.DataFrameStatFunctions: Functionality for statistic functions with DataFrame.

The RDD supports two types of operations, transformation and action. Table 1 and Table 2 show the common methods.

**Table 1** Transformation
Method	Description
map(f, preservesPartitioning=False)	Return a new RDD by applying a function to all elements of this RDD.
filter(f)	Return a new RDD containing only the elements that satisfy a predicate.
flatMap(f, preservesPartitioning=False)	Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
sample(withReplacement, fraction, seed=None)	Return a sampled subset of this RDD.
union(rdds)	Return a new RDD, contains source RDD and the group of RDD's elements.
distinct([numPartitions: Int]): RDD[T]	Return a new RDD containing the distinct elements in this RDD.
groupByKey(): RDD[(K, Iterable[V])]	Group the values for each key in the RDD into a single sequence. Hash-partitions the resulting RDD with into numPartitions partitions.
reduceByKey(func, numPartitions=None)	Merge the values for each key using an associative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce. Output will be hash-partitioned with numPartitions partitions.
sortByKey(ascending=True, numPartitions=None, keyfunc=function <lambda>)	Sort the RDD by key, so that each partition contains a sorted range of the elements. Calling collect or save on the resulting RDD will return or output an ordered list of records (in the save case, they will be written to multiple part-X files in the filesystem, in order of the keys).
join(other, numPartitions)	Return an RDD containing all pairs of elements with matching keys in this and other. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this and (k, v2) is in other. Performs a hash join across the cluster.
cogroup(other, numPartitions)	For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cartesian(other)	Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.

**Table 2** Action
Method	Description
reduce(f)	Reduces the elements of this RDD using the specified commutative and associative binary operator.
collect()	Return an array that contains all of the elements in this RDD.
count()	Return the number of elements in the RDD.
first()	Return the first element in this RDD.
take(num)	Take the first num elements of the RDD. This currently scans the partitions one by one, so it will be slow if a lot of partitions are required. In that case, use collect() to get the whole RDD instead.
takeSample(withReplacement, num, seed)	Return a fixed-size sampled subset of this RDD in an array
saveAsTextFile(path, compressionCodecClass)	Save this RDD as a compressed text file, using string representations of elements.
saveAsSequenceFile(path, compressionCodecClass=None)	Extra functions available on RDDs of (key, value) pairs to create a Hadoop SequenceFile, through an implicit conversion.
countByKey()	Count the appearance times of each key.
foreach(func)	Applies a function f to all elements of this RDD.
countByValue()	Return the count of each unique value in this RDD as a map of (value, count) pairs. The final combine step happens locally on the master, equivalent to running a single reduce task.

Spark Streaming Common Interfaces

Spark Streaming mainly uses the following classes:

pyspark.streaming.StreamingContext: A StreamingContext is the main entry point for Spark Streaming functionality. Besides the basic information (such as, cluster URL and job name) to internally create a SparkContext, it provides methods used to create DStreams from various input sources.
pyspark.streaming.DStream: A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous sequence of RDDs (of the same type) representing a continuous stream of data.
dstream.PariDStreamFunctions: the Dstream of key-value, common operations are groupByKey and reduceByKey.
The cooperated Java APIs of Spark Streaming are JavaStreamingContext, JavaDStream, JavaPairDStream.

Common methods of Spark Streaming are the same as those of Spark Core. The following table describes some special Spark Streaming methods.

**Table 3** Spark Streaming common interfaces
Method	Description
socketTextStream(hostname, port, storageLevel)	Reduces the elements of this RDD using the specified commutative and associative binary operator.
start()	Return an array that contains all of the elements in this RDD.
awaitTermination(timeout)	Return the number of elements in the RDD.
stop(stopSparkContext, stopGraceFully)	Return the first element in this RDD.
UpdateStateByKey(func)	Take the first num elements of the RDD. This currently scans the partitions one by one, so it will be slow if a lot of partitions are required. In that case, use collect() to get the whole RDD instead.
window(windowLength, slideInterval)	Return a fixed-size sampled subset of this RDD in an array
countByWindow(windowLength, slideInterval)	Save this RDD as a compressed text file, using string representations of elements.
reduceByWindow(func, windowLength, slideInterval)	Count the appearance times of each key.
join(other,numPartitions)	Applies a function f to all elements of this RDD. Return the count of each unique value in this RDD as a map of (value, count) pairs. The final combine step happens locally on the master, equivalent to running a single reduce task.

SparkSQL Common Interfaces

Spark SQL mainly uses the following classes:

pyspark.sql.SQLContext: main entrance of the Spark SQL function and DataFrame.
pyspark.sql.DataFrame: a distributed dataset organized by naming columns.
pyspark.sql.HiveContext: A variant of Spark SQL that integrates with data stored in Hive.
pyspark.sql.DataFrameStatFunctions: Functionality for statistic functions with DataFrame.
pyspark.sql.functions: A collection of builtin functions.
pyspark.sql.Window: Utility functions for defining window in DataFrames.

**Table 4** Spark SQL common Actions
Method	Description
collect()	Return an array containing all DataFrame columns.
count()	Return the number of DataFrame rows.
describe()	Count the statistic information, including the counting, average value, standard deviation, minimum value and maximum value.
first()	Return the first row.
head(n)	Return the first n rows.
show()	Display the first 20 rows in table.
take(num)	Return the first n rows in the DataFrame.

**Table 5** Basic DataFrame Functions
Method	Description
explain()	Print the logical plan and physical plan of the SQL.
printSchema()	Print the schema information to the console.
registerTempTable(name)	Register the DataFrame as a temporary table, whose period is bound to the SQLContext.
toDF()	Return a DataFrame whose columns are renamed.