Spark Python APIs

Updated on 2024-10-23 GMT+08:00

View PDF

To avoid API compatibility or reliability issues after updates to the open-source Spark, it is advisable to use APIs of the version you are currently using.

Spark Core Common Interfaces

Spark mainly uses the following classes:

pyspark.SparkContext: external API of Spark. It provides the functions of Spark for Python applications that invoke this class, for example, connecting Spark clusters, creating RDDs, and broadcasting variables.
pyspark.SparkConf: Spark application configuration class. It is used to set an application name, execution mode, and executor memory.
pyspark.RDD: class used to define the RDD in the Spark application. The class provides the data collection operation methods, such as map and filter.
pyspark.Broadcast: broadcast variable class. This class retains one read-only variable, and caches it on each machine, instead of saving a copy for each task.
pyspark.StorageLevel: data storage levels, including memory (MEMORY_ONLY), disk (DISK_ONLY), and memory+disk (MEMORY_AND_DISK).
pyspark.sql.SQLContext: Main entry point for SparkSQL functionality. It can be used to create DataFrame, register DataFrame as a table, and execute SQL on a table.
pyspark.sql.DataFrame: A distributed collection of data grouped into named columns. A DataFrame is equivalent to a relational table in Spark SQL, and can be created using various functions in SQLContext.
pyspark.sql.DataFrameNaFunctions: Functionality for working with missing data in DataFrame.
pyspark.sql.DataFrameStatFunctions: Functionality for statistic functions with DataFrame.

The RDD supports two types of operations, transformation and action. Table 1 and Table 2 show the common methods.

**Table 1** Transformation
Method	Description
map(f, preservesPartitioning=False)	Returns a new RDD by applying a function to all elements of this RDD.
filter(f)	Invokes the Func method for all RDD elements to generate a satisfied data set that is returned in the form of RDD.
flatMap(f, preservesPartitioning=False)	Returns a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
sample(withReplacement, fraction, seed=None)	Returns a sampled subset of this RDD.
union(rdds)	Returns a new RDD, contains source RDD and the group of RDD's elements.
distinct([numPartitions: Int]): RDD[T]	Returns a new RDD containing the distinct elements in this RDD.
groupByKey(): RDD[(K, Iterable[V])]	Returns (K,Iterable[V]) and combines the values of the same key to a set.
reduceByKey(func, numPartitions=None)	Invokes Func on the values of the same key.
sortByKey(ascending=True, numPartitions=None, keyfunc=function <lambda>)	Sorts by key in ascending or descending order. Ascending is of the boolean type.
join(other, numPartitions)	Returns the dataset of (K,(V,W)) when the (K,V) and (K,W) datasets exist. numPartitions indicates the number of concurrent tasks.
cogroup(other, numPartitions)	Returns the dataset of (K, (Iterable[V], Iterable[W])) when the (K,V) and (K,W) datasets of two key-value pairs exist. numPartitions indicates the number of concurrent tasks.
cartesian(other)	Returns the Cartesian product of the RDD and other RDDs.

**Table 2** Action
API	Description
reduce(f)	Invokes Func on elements of the RDD.
collect()	Returns an array that contains all of the elements in this RDD.
count()	Returns the number of elements in the dataset.
first()	Returns the first element in the dataset.
take(num)	Returns the first num elements of the RDD.
takeSample(withReplacement, num, seed)	Samples the dataset randomly and returns a dataset of num elements. withReplacement indicates whether replacement is used.
saveAsTextFile(path, compressionCodecClass)	Writes the dataset to a text file, HDFS, or file system supported by HDFS. Spark converts each record to a row of records and then writes it to the file.
saveAsSequenceFile(path, compressionCodecClass=None)	This API can be used only on the key-value pair, and then it generates SequenceFile and writes the file to the local or Hadoop file system.
countByKey()	Counts the appearance times of each key.
foreach(func)	Applies a function f to all elements of this RDD.
countByValue()	Counts the times that each value of the RDD occurs.

Spark Streaming Common Interfaces

Spark Streaming mainly uses the following classes:

pyspark.streaming.StreamingContext: main entrance of Spark Streaming. It provides methods for creating the DStream. A batch interval needs to be set in the input parameter.
pyspark.streaming.DStream: A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous sequence of RDDs (of the same type) representing a continuous stream of data.
dstream.PariDStreamFunctions: Dstream of key-value, common operations are groupByKey and reduceByKey.
The cooperated Java API of Spark Streaming are JavaStreamingContext, JavaDStream, JavaPairDStream.

Common methods of Spark Streaming are the same as those of Spark Core. The following table describes some special Spark Streaming methods.

**Table 3** Spark Streaming common interfaces
Method	Description
socketTextStream(hostname, port, storageLevel)	Creates an input stream from the TCP source host:port.
start()	Starts the Spark Streaming computing.
awaitTermination(timeout)	Terminates the await of the process, which is similar to pressing Ctrl+C.
stop(stopSparkContext, stopGraceFully)	Stops Spark Streaming computing. stopSparkContext is used to determine whether SparkContext needs to be terminated. StopGracefully is used to determine whether to wait for all the received data to be processed.
UpdateStateByKey(func)	Updates the status of DStream. To use this method, you need to define the state and state update functions.
window(windowLength, slideInterval)	Generates a new DStream by batch calculating according to the window of the source DStream.
countByWindow(windowLength, slideInterval)	Returns the number of sliding window elements in the stream.
reduceByWindow(func, windowLength, slideInterval)	When the key-value pair of DStream is invoked, a new key-value pair of DStream is returned. The value of each key is obtained by aggregating the reduce function in batches in the sliding window.
join(other,numPartitions)	Performs a join operation between different Spark Streamings.

SparkSQL Common Interfaces

Spark SQL mainly uses the following classes:

pyspark.sql.SQLContext: main entrance of the Spark SQL function and DataFrame.
pyspark.sql.DataFrame: a distributed dataset organized by naming columns.
pyspark.sql.HiveContext: main entrance for obtaining data stored in Hive.
pyspark.sql.DataFrameStatFunctions: Functionality for statistic functions with DataFrame.
pyspark.sql.functions: A collection of builtin functions.
pyspark.sql.Window: window function provided by SQL

**Table 4** Spark SQL common Actions
Method	Description
collect()	Returns an array containing all DataFrame columns.
count()	Returns the number of DataFrame rows.
describe()	Counts the statistic information, including the counting, average value, standard deviation, minimum value and maximum value.
first()	Returns the first row.
head(n)	Returns the first n rows.
show()	Displays DataFrame in a table.
take(num)	Returns the first num rows in the DataFrame.

**Table 5** Basic DataFrame Functions
Method	Description
explain()	Prints the logical plan and physical plan of the SQL.
printSchema()	Prints the schema information to the console.
registerTempTable(name)	Registers the DataFrame as a temporary table, whose period is bound to the SQLContext.
toDF()	Returns a DataFrame whose columns are renamed.