Developing an MRS Spark Python Job

This section describes how to develop an MRS Spark Python on DataArts Factory.

• Prepare the script file wordcount.py with the following content:

  # -*- coding: utf-8 -*
  import sys
  from pyspark import SparkConf, SparkContext
  def show(x):
      print(x)
  if __name__ == "__main__":
      if len(sys.argv) < 2:
          print ("Usage: wordcount <inputPath> <outputPath>")
          exit(-1)
      # Create SparkConf.
      conf = SparkConf().setAppName("wordcount")
      # Create SparkContext. Pass the conf=conf parameter.
      sc = SparkContext(conf=conf)
      inputPath = sys.argv[1]
      outputPath = sys.argv[2]
      lines = sc.textFile(name = inputPath)
      # Split each line of data by space to obtain words.
      words = lines.flatMap(lambda line:line.split(" "),True)
      # Pair each word into a tuple count 1.
      pairWords = words.map(lambda word:(word,1),True)
      # Use three partitions (reduceByKey) for summarization.
      result = pairWords.reduceByKey(lambda v1,v2:v1+v2)
      # Print the result.
      result.foreach(lambda t :show(t))
      # Save the result to a file.
      result.saveAsTextFile(outputPath)
      # Stop SparkContext.
      sc.stop()

  

  The encoding format must be set to UTF-8. Otherwise, an error will occur during script execution.

• Prepare the data file in.txt, which contains some English words.

Procedure

1. Upload the script and data file to the OBS bucket.
   Figure 1 Uploading files to an OBS bucket
   
   

   In this example, upload wordcount.py and in.txt to obs://obs-tongji/python/.

2. Create an empty job named job_MRS_Spark_Python.
   Figure 2 Creating a job
   

3. Go to the job development page, drag the MRS Spark Python node to the canvas, and click the node to configure its properties.
   Figure 3 Configuring properties for an MRS Spark Python node
   

   Parameter descriptions:

   --master
   yarn
   --deploy-mode
   cluster
   obs://obs-tongji/python/wordcount.py
   obs://obs-tongji/python/in.txt
   obs://obs-tongji/python/out

   Specifically:

   obs://obs-tongji/python/wordcount.py is the directory where the script is stored.

   obs://obs-tongji/python/in.txt is the directory where the wordcount.py parameters are passed. You can pass the words to count.

   obs://obs-tongji/python/out is the directory where output parameters are stored. This directory will also be created in the OBS bucket automatically. If the out directory already exists in the OBS bucket, an error will occur.

4. Click Test to execute the script job.
5. After the test is complete, click Submit.
6. Choose Monitor Job in the navigation pane and view the job execution result.
   Figure 4 Viewing the job execution result
   

   The job log shows that the job was successfully executed.

   Figure 5 Job run logs
   
   Figure 6 Job execution status
   

7. View the returned records in the OBS bucket. (Skip this step if the return function is not configured.)
   Figure 7 Viewing the returned records in the OBS bucket
   

from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName("master"). setMaster("yarn")
sc = SparkContext(conf=conf)
print("hello python")
sc.stop()