Help Center/ MapReduce Service/ Developer Guide (LTS)/ Spark2x Development Guide (Common Mode)/ Preparing a Spark Application Development Environment/ Configuring the Spark Python3 Sample Project
Updated on 2024-08-10 GMT+08:00
View PDF
Share

Configuring the Spark Python3 Sample Project

Overview

To run the Python3 API sample code of the Spark2x component of MRS, perform the following operations.

Procedure

  1. Install Python3 of 3.6 or a higher version on the client.

    The Python version can be viewed by running the python3 command on the CLI of the client. The following information indicates that the Python version is 3.8.2. 
    Python 3.8.2 (default, Jun 23 2020, 10:26:03)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.

  2. Setuptools 47.3.1 must be installed on the client.

    To obtain the software, visit the official websites.

    https://pypi.org/project/setuptools/#files

    Copy the downloaded setuptools package to the client, decompress the package, go to the decompressed directory, and run the python3 setup.py install command in the CLI of the client.

    The following information indicates that setuptools 47.3.1 is installed successfully.

    Finished processing dependencies for setuptools==47.3.1

  3. Install Python on the client.

    1. Obtain the sample project folder python3-examples in the src\hive-examples directory where the sample code is decompressed. For details, see Obtaining the MRS Application Development Sample Project.
    2. Go to the python3-examples folder.
    3. Go to the dependency_python3.6, dependency_python3.7, or dependency_python3.8 folder based on the Python3 version.
    4. Run the whereis easy_install command to find the path of the easy_install program. If there are multiple paths, run the easy_install --version command to select the easy_install path corresponding to the setuptools version, for example, /usr/local/bin/easy_install.
    5. Run the easy_install command to install the EGG files in the dependency_python3.x folder in sequence. Example:

      /usr/local/bin/easy_install future-0.18.2-py3.8.egg

      If the following information is displayed, the EGG file is successfully installed.

      Finished processing dependencies for future==0.18.2