Java

To avoid API compatibility or reliability issues after updates to the open-source Spark, it is advisable to use APIs of the version you are currently using.

Spark Core Common Interfaces

Spark mainly uses the following classes:

JavaSparkContext: external interface of Spark, which is used to provide the functions of Spark for Java applications that invoke this class, for example, connecting Spark clusters and generating RDDs, accumulations, and broadcasts. It functions as a container.
SparkConf: Spark application configuration class, which is used to configure the application name, execution model, and executor memory.
JavaRDD: class used to define the JavaRDD in the Java application, which functions like the RDD (Resilient Distributed Dataset) class of Scala.
JavaPairRDD: indicates the JavaRDD in the key-value format. This class provides methods such as groupByKey and reduceByKey.

Broadcast: broadcast variable class. This class retains one read-only variable, and caches it on each machine, instead of saving a copy for each task.
StorageLevel: data storage level class, including MEMORY_ONLY, DISK_ONLY, and MEMORY_AND_DISK.

The JavaRDD supports two types of operations, transformation and action. Table 1 and Table 2 show the common methods.

**Table 1** Transformation
Method	Description
<R> JavaRDD<R> map(Function<T,R> f)	Return a new RDD by applying a function to all elements of this RDD.
JavaRDD<T> filter(Function<T,Boolean> f)	Return a new RDD containing only the elements that satisfy a predicate.
<U> JavaRDD<U> flatMap(FlatMapFunction<T,U> f)	Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
JavaRDD<T> sample(boolean withReplacement, double fraction, long seed)	Return a sampled subset of this RDD.
JavaRDD<T> distinct(int numPartitions)	Return a new RDD containing the distinct elements in this RDD.
JavaPairRDD<K,Iterable<V>> groupByKey(int numPartitions)	Group the values for each key in the RDD into a single sequence. Hash-partitions the resulting RDD with into numPartitions partitions.
JavaPairRDD<K,V> reduceByKey(Function2<V,V,V> func, int numPartitions)	Merge the values for each key using an associative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce. Output will be hash-partitioned with numPartitions partitions.
JavaPairRDD<K,V> sortByKey(boolean ascending, int numPartitions)	Sort the RDD by key, so that each partition contains a sorted range of the elements. Calling collect or save on the resulting RDD will return or output an ordered list of records (in the save case, they will be written to multiple part-X files in the filesystem, in order of the keys).
JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other)	Return an RDD containing all pairs of elements with matching keys in this and other. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this and (k, v2) is in other. Performs a hash join across the cluster.
JavaPairRDD<K,scala.Tuple2<Iterable<V>,Iterable<W>>> cogroup(JavaPairRDD<K,W> other, int numPartitions)	For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
JavaPairRDD<T,U> cartesian(JavaRDDLike<U,?> other)	Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.

**Table 2** Action
Method	Description
T reduce(Function2<T,T,T> f)	Reduces the elements of this RDD using the specified commutative and associative binary operator.
java.util.List<T> collect()	Return an array that contains all of the elements in this RDD.
long count()	Return the number of elements in the RDD.
T first()	Return the first element in this RDD.
java.util.List<T> take(int num)	Take the first num elements of the RDD. This currently scans the partitions one by one, so it will be slow if a lot of partitions are required. In that case, use collect() to get the whole RDD instead.
java.util.List<T> takeSample(boolean withReplacement, int num, long seed)	Return a fixed-size sampled subset of this RDD in an array
void saveAsTextFile(String path, Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec)	Save this RDD as a compressed text file, using string representations of elements.
java.util.Map<K,Object> countByKey()	Count the appearance times of each key.
void foreach(VoidFunction<T> f)	Applies a function f to all elements of this RDD.
java.util.Map<T,Long> countByValue()	Return the count of each unique value in this RDD as a map of (value, count) pairs. The final combine step happens locally on the master, equivalent to running a single reduce task.

**Table 3** New APIs of Spark core
API	Description
public java.util.concurrent.atomic.AtomicBoolean isSparkContextDown()	Determines whether sparkContext is shut down completely. The initial value is false. The value true indicates that sparkContext is shut down completely. The value false indicates that sparkContext is not shut down. For example, jsc.sc().isSparkContextDown().get() == true indicates that sparkContext is shut down completely.

Spark Streaming Common Interfaces

Spark Streaming mainly uses the following classes:

JavaStreamingContext: main entrance of the Spark Streaming, which is used to provide methods for creating the DStream. Intervals by batch need to be set in the input parameter.
JavaDStream: a type of data which indicates the RDDs continuous sequence. It indicates the continuous data flow.
JavaPairDStream: interface of KV DStream, which is used to provide the reduceByKey and join operations.
JavaReceiverInputDStream<T>: specifies any inflow accepting data from the network.

Common methods of Spark Streaming are the same as those of Spark Core. The following table describes some special Spark Streaming methods.

**Table 4** Spark Streaming methods
Method	Description
JavaReceiverInputDStream<java.lang.String> socketStream(java.lang.String hostname,int port)	It creates an inflow to receive data from the corresponding hostname and port through the TCP socket. Received data is resolved to the UTF8 format. The default storage level is the Memory+Disk.
JavaDStream<java.lang.String> textFileStream(java.lang.String directory)	It creates an inflow to detect new files compatible with the Hadoop file system, and read it as a text file. The directory of the input parameter is an HDFS directory.
void start()	It starts the Spark Streaming calculation.
void awaitTermination()	It terminates the await of the process, which is similar to pressing Ctrl+C.
void stop()	It stops the Spark Streaming calculation.
<T> JavaDStream<T> transform(java.util.List<JavaDStream<?>> dstreams,Function2<java.util.List<JavaRDD<?>>,Time,JavaRDD<T>> transformFunc)	It performs the Function operation on each RDD to obtain a new DStream. In this function, the sequence of the JavaRDDs must be the same as the corresponding DStreams.
<T> JavaDStream<T> union(JavaDStream<T> first,java.util.List<JavaDStream<T>> rest)	It creates a unified Dstream from multiple DStreams with the same type and sliding window.

**Table 5** Spark Streaming enhancement interface
Method	Description
JAVADStreamKafkaWriter.writeToKafka()	Writes data from the DStream into Kafka in batch.
JAVADStreamKafkaWriter.writeToKafkaBySingle()	Writes data from DStream into Kafka one by one.

Spark SQL Common Interfaces

Spark SQL mainly uses the following classes:

SQLContext: main entrance of the Spark SQL function and DataFrame.
DataFrame: a distributed dataset organized by naming columns.
DataFrameReader: interface for loading the DataFrame from external storage systems.
DataFrameStatFunctions: implementation the statistic function of the DataFrame.
UserDefinedFunction: function defined by users.

Common Actions methods are described in the following table.

**Table 6** Spark SQL methods
Method	Description
Row[] collect()	Return an array containing all DataFrame columns.
long count()	Return the number of DataFrame rows.
DataFrame describe(java.lang.String... cols)	Count the statistic information, including the counting, average value, standard deviation, minimum value and maximum value.
Row first()	Return the first row.
Row[] head(int n)	Return the first n rows.
void show()	Display the first 20 rows in table.
Row[] take(int n)	Return the first n rows in the DataFrame.

**Table 7** Basic DataFrame Functions
Method	Description
void explain(boolean extended)	Print the logical plan and physical plan of the SQL.
void printSchema()	Print the schema information to the console.
registerTempTable	Register the DataFrame as a temporary table, whose period is bound to the SQLContext.
DataFrame toDF(java.lang.String... colNames)	Return a DataFrame whose columns are renamed.
DataFrame sort(java.lang.String sortCol,java.lang.String... sortCols)	Based on different columns, sort columns in ascending or descending orders.
GroupedData rollup(Column... cols)	Perform multiple-dimension crankback on the specified columns in the DataFrame.