El contenido no se encuentra disponible en el idioma seleccionado. Estamos trabajando continuamente para agregar más idiomas. Gracias por su apoyo.

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Spark Java APIs

Updated on 2024-08-10 GMT+08:00

To avoid API compatibility or reliability issues after updates to the open-source Spark, it is advisable to use APIs of the version you are currently using.

Common Spark Core APIs

Spark mainly uses the following classes:

  • JavaSparkContext: external interface of Spark, which is used to provide the functions of Spark for Java applications that invoke this class, for example, connecting Spark clusters and generating RDDs, accumulations, and broadcasts. It functions as a container.
  • SparkConf: Spark application configuration class, which is used to configure the application name, execution model, and executor memory.
  • JavaRDD: class used to define the JavaRDD in the Java application, which functions like the RDD class of Scala.
  • JavaPairRDD: JavaRDD in the key-value format. This class provides methods such as groupByKey and reduceByKey.
  • Broadcast: broadcast variable class. This class retains one read-only variable, and caches it on each machine, instead of saving a copy for each task.
  • StorageLevel: data storage levels, including memory (MEMORY_ONLY), disk (DISK_ONLY), and memory+disk (MEMORY_AND_DISK).

The JavaRDD supports two types of operations, transformation and action. Table 1 and Table 2 show the common methods.

Table 1 Transformation

Method

Description

<R> JavaRDD<R> map(Function<T,R> f)

Returns a new RDD by applying a function to all elements of this RDD.

JavaRDD<T> filter(Function<T,Boolean> f)

Invokes Function on all elements of the RDD and returns the element that is true.

<U> JavaRDD<U> flatMap(FlatMapFunction<T,U> f)

Returns a new RDD by first applying a function to all elements of this RDD, and then flattening the results.

JavaRDD<T> sample(boolean withReplacement, double fraction, long seed)

Takes samples.

JavaRDD<T> distinct(int numPartitions)

Returns a new RDD containing the distinct elements in this RDD.

JavaPairRDD<K,Iterable<V>> groupByKey(int numPartitions)

Returns (K,Seq[V]) and combines the values of the same key to a set.

JavaPairRDD<K,V> reduceByKey(Function2<V,V,V> func, int numPartitions)

Invokes a function for values with the same key.

JavaPairRDD<K,V> sortByKey(boolean ascending, int numPartitions)

Sorts by key in ascending or descending order. Ascending is of the boolean type.

JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other)

Returns the dataset of (K,(V,W)) when the (K,V) and (K,W) datasets exist. numTasks indicates the number of concurrent tasks.

JavaPairRDD<K,scala.Tuple2<Iterable<V>,Iterable<W>>> cogroup(JavaPairRDD<K,W> other, int numPartitions)

Returns the dataset of <K,scala.Tuple2<Iterable<V>,Iterable<W>>> when the (K,V) and (K,W) datasets exist. numTasks indicates the number of concurrent tasks.

JavaPairRDD<T,U> cartesian(JavaRDDLike<U,?> other)

Returns the Cartesian product of the RDD and other RDDs.

Table 2 Action

Method

Description

T reduce(Function2<T,T,T> f)

Invokes Function2 on elements of the RDD.

java.util.List<T> collect()

Returns an array that contains all of the elements in this RDD.

long count()

Returns the number of elements in the dataset.

T first()

Returns the first element in the dataset.

java.util.List<T> take(int num)

Returns the first n elements in this RDD.

java.util.List<T> takeSample(boolean withReplacement, int num, long seed)

Samples the dataset randomly and returns a dataset of num elements. withReplacement indicates whether replacement is used.

void saveAsTextFile(String path, Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec)

Writes the dataset to a text file, HDFS, or file system supported by HDFS. Spark converts each record to a row of records and then writes it to the file.

java.util.Map<K,Object> countByKey()

Counts the appearance times of each key.

void foreach(VoidFunction<T> f)

Runs func on each element of the dataset.

java.util.Map<T,Long> countByValue()

Counts the times that each element of the RDD occurs.

Table 3 New APIs of Spark core

API

Description

public java.util.concurrent.atomic.AtomicBoolean isSparkContextDown()

Determines whether sparkContext is shut down completely. The initial value is false.

The value true indicates that sparkContext is shut down completely.

The value false indicates that sparkContext is not shut down.

For example, jsc.sc().isSparkContextDown().get() == true indicates that sparkContext is shut down completely.

Common Spark Streaming APIs

Spark Streaming mainly uses the following classes:

  • JavaStreamingContext: main entrance of Spark Streaming. It provides methods for creating the DStream. A batch interval needs to be set in the input parameter.
  • JavaDStream: a type of data which indicates the RDDs continuous sequence. It indicates the continuous data flow.
  • JavaPairDStream: API of KV DStream, which is used to provide the reduceByKey and join operations.
  • JavaReceiverInputDStream<T>: specifies any inflow accepting data from the network.

The techniques used in Spark Streaming are comparable to those in Spark Core. Check out the table below for some examples of Spark Streaming methods.

Table 4 Spark Streaming methods

Method

Description

JavaReceiverInputDStream<java.lang.String> socketStream(java.lang.String hostname,int port)

Creates an inflow to receive data from the corresponding hostname and port through the TCP socket. Received data is resolved to the UTF8 format. The default storage level is the Memory+Disk.

JavaDStream<java.lang.String> textFileStream(java.lang.String directory)

Creates an inflow to detect new files compatible with the Hadoop file system, and read it as a text file. The directory of the input parameter is an HDFS directory.

void start()

Starts the Spark Streaming computing.

void awaitTermination()

Terminates the await of the process, which is similar to pressing Ctrl+C.

void stop()

Stops the Spark Streaming computing.

<T> JavaDStream<T> transform(java.util.List<JavaDStream<?>> dstreams,Function2<java.util.List<JavaRDD<?>>,Time,JavaRDD<T>> transformFunc)

Performs the Function operation on each RDD to obtain a new DStream. In this function, the sequence of the JavaRDDs must be the same as the corresponding DStreams.

<T> JavaDStream<T> union(JavaDStream<T> first,java.util.List<JavaDStream<T>> rest)

Creates a unified DStream from multiple DStreams with the same type and sliding window.

Table 5 Spark Streaming enhanced feature APIs

API

Description

JAVADStreamKafkaWriter.writeToKafka()

Writes data from DStream into Kafka in batch.

JAVADStreamKafkaWriter.writeToKafkaBySingle()

Writes data from DStream into Kafka one by one.

Common Spark SQL APIs

Spark SQL mainly uses the following classes:

  • SQLContext: main entrance of Spark SQL functions and DataFrame.
  • DataFrame: distributed dataset organized by naming columns.
  • DataFrameReader: API for loading a DataFrame from external storage systems.
  • DataFrameStatFunctions: implements the statistics function of a DataFrame.
  • UserDefinedFunction: function defined by users.

The following table provides common actions methods.

Table 6 Spark SQL methods

Method

Description

Row[] collect()

Returns an array containing all DataFrame columns.

long count()

Returns the number of rows in a DataFrame.

DataFrame describe(java.lang.String... cols)

Counts the statistic information, including the counting, average value, standard deviation, minimum value and maximum value.

Row first()

Returns the first row.

Row[] head(int n)

Returns the first n rows.

void show()

Displays the first 20 rows in a DataFrame using a table.

Row[] take(int n)

Returns the first n rows in the DataFrame.

Table 7 Basic DataFrame Functions

Function

Description

void explain(boolean extended)

Prints the logical plan and physical plan of the SQL.

void printSchema()

Prints the schema information to the console.

registerTempTable

Registers the DataFrame as a temporary table, whose period is bound to the SQLContext.

DataFrame toDF(java.lang.String... colNames)

Returns a DataFrame whose columns are renamed.

DataFrame sort(java.lang.String sortCol,java.lang.String... sortCols)

Sorts columns in ascending or descending orders based on different columns.

GroupedData rollup(Column... cols)

Performs multiple-dimension crankback on the specified columns in a DataFrame.

Utilizamos cookies para mejorar nuestro sitio y tu experiencia. Al continuar navegando en nuestro sitio, tú aceptas nuestra política de cookies. Descubre más

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback