Spark Java APIs

To avoid API compatibility or reliability issues after updates to the open-source Spark, it is advisable to use APIs of the version you are currently using.

Spark Core Common Interfaces

Spark mainly uses the following classes:

JavaSparkContext: external interface of Spark, which is used to provide the functions of Spark for Java applications that invoke this class, for example, connecting Spark clusters and generating RDDs, accumulations, and broadcasts. It functions as a container.
SparkConf: Spark application configuration class, which is used to configure the application name, execution model, and executor memory.
JavaRDD: class used to define the JavaRDD in the Java application, which functions like the RDD (Resilient Distributed Dataset) class of Scala.
JavaPairRDD: indicates the JavaRDD in the key-value format. This class provides methods such as groupByKey and reduceByKey.

Broadcast: broadcast variable class. This class retains one read-only variable, and caches it on each machine, instead of saving a copy for each task.
StorageLevel: data storage levels, including memory (MEMORY_ONLY), disk (DISK_ONLY), and memory+disk (MEMORY_AND_DISK).

The JavaRDD supports two types of operations, transformation and action. Table 1 and Table 2 show the common methods.

**Table 1** Transformation
Method	Description
<R> JavaRDD<R> map(Function<T,R> f)	Returns a new RDD by applying a function to all elements of this RDD.
JavaRDD<T> filter(Function<T,Boolean> f)	Invokes Function on all elements of the RDD and returns the element that is true.
<U> JavaRDD<U> flatMap(FlatMapFunction<T,U> f)	Returns a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
JavaRDD<T> sample(boolean withReplacement, double fraction, long seed)	Returns a sampled subset of this RDD.
JavaRDD<T> distinct(int numPartitions)	Returns a new RDD containing the distinct elements in this RDD.
JavaPairRDD<K,Iterable<V>> groupByKey(int numPartitions)	Returns (K,Seq[V]) and combines the values of the same key to a set.
JavaPairRDD<K,V> reduceByKey(Function2<V,V,V> func, int numPartitions)	Invokes Function on the values of the same key.
JavaPairRDD<K,V> sortByKey(boolean ascending, int numPartitions)	Sorts by key in ascending or descending order. Ascending is of the boolean type.
JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other)	Returns the dataset of (K,(V,W)) when the (K,V) and (K,W) datasets exist. numTasks indicates the number of concurrent tasks.
JavaPairRDD<K,scala.Tuple2<Iterable<V>,Iterable<W>>> cogroup(JavaPairRDD<K,W> other, int numPartitions)	Returns the dataset of <K,scala.Tuple2<Iterable<V>,Iterable<W>>> when the (K,V) and (K,W) datasets exist. numTasks indicates the number of concurrent tasks.
JavaPairRDD<T,U> cartesian(JavaRDDLike<U,?> other)	Returns the Cartesian product of the RDD and other RDDs.

**Table 2** Action
Method	Description
T reduce(Function2<T,T,T> f)	Invokes Function2 on elements of the RDD.
java.util.List<T> collect()	Returns an array that contains all of the elements in this RDD.
long count()	Returns the number of elements in the RDD.
T first()	Returns the first element in this RDD.
java.util.List<T> take(int num)	Returns the first n elements in this RDD.
java.util.List<T> takeSample(boolean withReplacement, int num, long seed)	Samples the dataset randomly and returns a dataset of num elements. withReplacement indicates whether replacement is used.
void saveAsTextFile(String path, Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec)	Writes the dataset to a text file, HDFS, or file system supported by HDFS. Spark converts each record to a row of records and then writes it to the file.
java.util.Map<K,Object> countByKey()	Counts the appearance times of each key.
void foreach(VoidFunction<T> f)	Applies a function f to all elements of this RDD.
java.util.Map<T,Long> countByValue()	Counts the times that each element of the RDD occurs.

**Table 3** New APIs of Spark core
API	Description
public java.util.concurrent.atomic.AtomicBoolean isSparkContextDown()	Determines whether sparkContext is shut down completely. The initial value is false. The value true indicates that sparkContext is shut down completely. The value false indicates that sparkContext is not shut down. For example, jsc.sc().isSparkContextDown().get() == true indicates that sparkContext is shut down completely.

Spark Streaming Common Interfaces

Spark Streaming mainly uses the following classes:

JavaStreamingContext: main entrance of the Spark Streaming, which is used to provide methods for creating the DStream. Intervals by batch need to be set in the input parameter.
JavaDStream: a type of data which indicates the RDDs continuous sequence. It indicates the continuous data flow.
JavaPairDStream: interface of KV DStream, which is used to provide the reduceByKey and join operations.
JavaReceiverInputDStream<T>: specifies any inflow accepting data from the network.

Common methods of Spark Streaming are the same as those of Spark Core. The following table describes some special Spark Streaming methods.

**Table 4** Spark Streaming methods
Method	Description
JavaReceiverInputDStream<java.lang.String> socketStream(java.lang.String hostname,int port)	Creates an inflow to receive data from the corresponding hostname and port through the TCP socket. Received data is resolved to the UTF8 format. The default storage level is the Memory+Disk.
JavaDStream<java.lang.String> textFileStream(java.lang.String directory)	Creates an inflow to detect new files compatible with the Hadoop file system, and read it as a text file. The directory of the input parameter is an HDFS directory.
void start()	Starts the Spark Streaming computing.
void awaitTermination()	Terminates the await of the process, which is similar to pressing Ctrl+C.
void stop()	Stops the Spark Streaming computing.
<T> JavaDStream<T> transform(java.util.List<JavaDStream<?>> dstreams,Function2<java.util.List<JavaRDD<?>>,Time,JavaRDD<T>> transformFunc)	Performs the Function operation on each RDD to obtain a new DStream. In this function, the sequence of the JavaRDDs must be the same as the corresponding DStreams.
<T> JavaDStream<T> union(JavaDStream<T> first,java.util.List<JavaDStream<T>> rest)	Creates a unified Dstream from multiple DStreams with the same type and sliding window.

**Table 5** Spark Streaming enhancement interface
Method	Description
JAVADStreamKafkaWriter.writeToKafka()	Writes data from the DStream into Kafka in batch.
JAVADStreamKafkaWriter.writeToKafkaBySingle()	Writes data from DStream into Kafka one by one.

Spark SQL Common Interfaces

Spark SQL mainly uses the following classes:

SQLContext: main entrance of the Spark SQL function and DataFrame.
DataFrame: a distributed dataset organized by naming columns.
DataFrameReader: interface for loading the DataFrame from external storage systems.
DataFrameStatFunctions: implementation the statistic function of the DataFrame.
UserDefinedFunction: function defined by users.

Common Actions methods are described in the following table.

**Table 6** Spark SQL methods
Method	Description
Row[] collect()	Returns an array containing all DataFrame columns.
long count()	Returns the number of DataFrame rows.
DataFrame describe(java.lang.String... cols)	Counts the statistic information, including the counting, average value, standard deviation, minimum value and maximum value.
Row first()	Returns the first row.
Row[] head(int n)	Returns the first n rows.
void show()	Displays the first 20 rows in table.
Row[] take(int n)	Returns the first n rows in the DataFrame.

**Table 7** Basic DataFrame Functions
Method	Description
void explain(boolean extended)	Prints the logical plan and physical plan of the SQL.
void printSchema()	Prints the schema information to the console.
registerTempTable	Registers the DataFrame as a temporary table, whose period is bound to the SQLContext.
DataFrame toDF(java.lang.String... colNames)	Returns a DataFrame whose columns are renamed.
DataFrame sort(java.lang.String sortCol,java.lang.String... sortCols)	Sorts columns in ascending or descending orders based on different columns.
GroupedData rollup(Column... cols)	Performs multiple-dimension crankback on the specified columns in the DataFrame.