Spark Java APIs

Updated on 2024-08-10 GMT+08:00

View PDF

To avoid API compatibility or reliability issues after updates to the open-source Spark, it is advisable to use APIs of the version you are currently using.

Common Spark Core APIs

Spark mainly uses the following classes:

JavaSparkContext: external interface of Spark, which is used to provide the functions of Spark for Java applications that invoke this class, for example, connecting Spark clusters and generating RDDs, accumulations, and broadcasts. It functions as a container.
SparkConf: Spark application configuration class, which is used to configure the application name, execution model, and executor memory.
JavaRDD: class used to define the JavaRDD in the Java application, which functions like the RDD class of Scala.
JavaPairRDD: JavaRDD in the key-value format. This class provides methods such as groupByKey and reduceByKey.

Broadcast: broadcast variable class. This class retains one read-only variable, and caches it on each machine, instead of saving a copy for each task.
StorageLevel: data storage levels, including memory (MEMORY_ONLY), disk (DISK_ONLY), and memory+disk (MEMORY_AND_DISK).

The JavaRDD supports two types of operations, transformation and action. Table 1 and Table 2 show the common methods.

**Table 1** Transformation
Method	Description
<R> JavaRDD<R> map(Function<T,R> f)	Returns a new RDD by applying a function to all elements of this RDD.
JavaRDD<T> filter(Function<T,Boolean> f)	Invokes Function on all elements of the RDD and returns the element that is true.
<U> JavaRDD<U> flatMap(FlatMapFunction<T,U> f)	Returns a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
JavaRDD<T> sample(boolean withReplacement, double fraction, long seed)	Takes samples.
JavaRDD<T> distinct(int numPartitions)	Returns a new RDD containing the distinct elements in this RDD.
JavaPairRDD<K,Iterable<V>> groupByKey(int numPartitions)	Returns (K,Seq[V]) and combines the values of the same key to a set.
JavaPairRDD<K,V> reduceByKey(Function2<V,V,V> func, int numPartitions)	Invokes a function for values with the same key.
JavaPairRDD<K,V> sortByKey(boolean ascending, int numPartitions)	Sorts by key in ascending or descending order. Ascending is of the boolean type.
JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other)	Returns the dataset of (K,(V,W)) when the (K,V) and (K,W) datasets exist. numTasks indicates the number of concurrent tasks.
JavaPairRDD<K,scala.Tuple2<Iterable<V>,Iterable<W>>> cogroup(JavaPairRDD<K,W> other, int numPartitions)	Returns the dataset of <K,scala.Tuple2<Iterable<V>,Iterable<W>>> when the (K,V) and (K,W) datasets exist. numTasks indicates the number of concurrent tasks.
JavaPairRDD<T,U> cartesian(JavaRDDLike<U,?> other)	Returns the Cartesian product of the RDD and other RDDs.

**Table 2** Action
Method	Description
T reduce(Function2<T,T,T> f)	Invokes Function2 on elements of the RDD.
java.util.List<T> collect()	Returns an array that contains all of the elements in this RDD.
long count()	Returns the number of elements in the dataset.
T first()	Returns the first element in the dataset.
java.util.List<T> take(int num)	Returns the first n elements in this RDD.
java.util.List<T> takeSample(boolean withReplacement, int num, long seed)	Samples the dataset randomly and returns a dataset of num elements. withReplacement indicates whether replacement is used.
void saveAsTextFile(String path, Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec)	Writes the dataset to a text file, HDFS, or file system supported by HDFS. Spark converts each record to a row of records and then writes it to the file.
java.util.Map<K,Object> countByKey()	Counts the appearance times of each key.
void foreach(VoidFunction<T> f)	Runs func on each element of the dataset.
java.util.Map<T,Long> countByValue()	Counts the times that each element of the RDD occurs.

**Table 3** New APIs of Spark core
API	Description
public java.util.concurrent.atomic.AtomicBoolean isSparkContextDown()	Determines whether sparkContext is shut down completely. The initial value is false. The value true indicates that sparkContext is shut down completely. The value false indicates that sparkContext is not shut down. For example, jsc.sc().isSparkContextDown().get() == true indicates that sparkContext is shut down completely.

Common Spark Streaming APIs

Spark Streaming mainly uses the following classes:

JavaStreamingContext: main entrance of Spark Streaming. It provides methods for creating the DStream. A batch interval needs to be set in the input parameter.
JavaDStream: a type of data which indicates the RDDs continuous sequence. It indicates the continuous data flow.
JavaPairDStream: API of KV DStream, which is used to provide the reduceByKey and join operations.
JavaReceiverInputDStream<T>: specifies any inflow accepting data from the network.

The techniques used in Spark Streaming are comparable to those in Spark Core. Check out the table below for some examples of Spark Streaming methods.

**Table 4** Spark Streaming methods
Method	Description
JavaReceiverInputDStream<java.lang.String> socketStream(java.lang.String hostname,int port)	Creates an inflow to receive data from the corresponding hostname and port through the TCP socket. Received data is resolved to the UTF8 format. The default storage level is the Memory+Disk.
JavaDStream<java.lang.String> textFileStream(java.lang.String directory)	Creates an inflow to detect new files compatible with the Hadoop file system, and read it as a text file. The directory of the input parameter is an HDFS directory.
void start()	Starts the Spark Streaming computing.
void awaitTermination()	Terminates the await of the process, which is similar to pressing Ctrl+C.
void stop()	Stops the Spark Streaming computing.
<T> JavaDStream<T> transform(java.util.List<JavaDStream<?>> dstreams,Function2<java.util.List<JavaRDD<?>>,Time,JavaRDD<T>> transformFunc)	Performs the Function operation on each RDD to obtain a new DStream. In this function, the sequence of the JavaRDDs must be the same as the corresponding DStreams.
<T> JavaDStream<T> union(JavaDStream<T> first,java.util.List<JavaDStream<T>> rest)	Creates a unified DStream from multiple DStreams with the same type and sliding window.

**Table 5** Spark Streaming enhanced feature APIs
API	Description
JAVADStreamKafkaWriter.writeToKafka()	Writes data from DStream into Kafka in batch.
JAVADStreamKafkaWriter.writeToKafkaBySingle()	Writes data from DStream into Kafka one by one.

Common Spark SQL APIs

Spark SQL mainly uses the following classes:

SQLContext: main entrance of Spark SQL functions and DataFrame.
DataFrame: distributed dataset organized by naming columns.
DataFrameReader: API for loading a DataFrame from external storage systems.
DataFrameStatFunctions: implements the statistics function of a DataFrame.
UserDefinedFunction: function defined by users.

The following table provides common actions methods.

**Table 6** Spark SQL methods
Method	Description
Row[] collect()	Returns an array containing all DataFrame columns.
long count()	Returns the number of rows in a DataFrame.
DataFrame describe(java.lang.String... cols)	Counts the statistic information, including the counting, average value, standard deviation, minimum value and maximum value.
Row first()	Returns the first row.
Row[] head(int n)	Returns the first n rows.
void show()	Displays the first 20 rows in a DataFrame using a table.
Row[] take(int n)	Returns the first n rows in the DataFrame.

**Table 7** Basic DataFrame Functions
Function	Description
void explain(boolean extended)	Prints the logical plan and physical plan of the SQL.
void printSchema()	Prints the schema information to the console.
registerTempTable	Registers the DataFrame as a temporary table, whose period is bound to the SQLContext.
DataFrame toDF(java.lang.String... colNames)	Returns a DataFrame whose columns are renamed.
DataFrame sort(java.lang.String sortCol,java.lang.String... sortCols)	Sorts columns in ascending or descending orders based on different columns.
GroupedData rollup(Column... cols)	Performs multiple-dimension crankback on the specified columns in a DataFrame.

Parent topic: Common Spark APIs

Previous topic: Common Spark APIs

Next topic: Spark Scala APIs

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot