Scala Sample Code

Updated on 2022-09-14 GMT+08:00

View PDF

Function Description

Collect statistics on female netizens who dwell on online shopping for more than 2 hours on the weekend.

Sample Code

The following code snippets are used as an example. For complete codes, see the com.huawei.bigdata.spark.examples.FemaleInfoCollection class.

// Configure the Spark application name.
val conf = new SparkConf().setAppName("CollectFemaleInfo")

// Submit a Spark job.
val sc = new SparkContext(conf)
// Reads data. The input parameter args(0) specifies the data path.
val text = sc.textFile(args(0))
// Filter data information of the time that female netizens spend online. 
val data = text.filter(_.contains("female"))
// Summarize the total time that each female netizen spends online. 
val femaleData:RDD[(String,Int)] = data.map{line =>
    val t= line.split(',')
    (t(0),t(2).toInt)
}.reduceByKey(_ + _)
// Filter the information of female netizens who spend more than 2 hours online and output the result.
val result = femaleData.filter(line => line._2 > 120)
result.foreach(println)

Parent topic: Spark Core Application

Previous topic: Java Sample Code

Next topic: Python Sample Code