Help Center/ MapReduce Service/ Developer Guide (LTS)/ Spark2x Development Guide (Security Mode)/ FAQs About Spark Application Development/ Using Conda to Package the Python Environment and Third-Party Libraries in the PySpark Program
Updated on 2026-08-27 GMT+08:00
View PDF
Share

Using Conda to Package the Python Environment and Third-Party Libraries in the PySpark Program

This section guides you to configure third-party Python libraries within the PySpark environment. Using other Python versions may cause compatibility issues. Using Conda to create a standalone Python environment allows you to flexibly install and manage third-party libraries, thereby avoiding conflicts with system Python or other project environments.

Preparations

Ensure that Conda has been installed in your development environment, which supports both Windows and Linux systems. You can download and install Conda from the Anaconda official website.

Creating and packaging a Conda environment

  • Create an environment.

    conda create -y -n pyspark_conda_env -c conda-forge conda-pack numpy python=3.9.0

  • Activate the environment.

    conda activate pyspark_conda_env

  • Install additional libraries.

    Run the conda install command after activating the environment.

  • Package the environment.

    conda pack -f -o pyspark_conda_env.tar.gz

  • Upload the package.

    Upload the generated pyspark_conda_env.tar.gz package to the distributed file system (such as HDFS) or object storage service (such as OBS) of your cluster.

Submitting a PySpark Job

Add the following parameters when submitting a PySpark job:

  • Specify a package. 
    --archives R_PATH/pyspark_conda_env.tar.gz#environment # R_PATH is the path of the package in the HDFS or OBS.
  • Configure the Python environment. 
    --archives R_PATH/pyspark_conda_env.tar.gz#environment # R_PATH is the path of the package in the HDFS or OBS.
# In local or yarn-client mode, upload the preceding package to the node where the job is submitted and decompress it.
--conf spark.pyspark.driver.python=L_PATH/pyspark_conda_env/bin/python # Do not set this parameter in cluster mode. L_PATH is the local directory where the package will be decompressed.
--conf spark.pyspark.python=./environment/bin/python
  • Configure for PySpark shell interactive jobs. 
    pyspark --archives R_PATH/pyspark_conda_env.tar.gz#environment
--conf spark.pyspark.driver.python=L_PATH/pyspark_conda_env/bin/python # L_PATH is the directory where pyspark_conda_env.tar.gz will be decompressed on this node.
--conf spark.pyspark.python=./environment/bin/python