Help Center/ MapReduce Service/ Best Practices/ Interconnection with Ecosystem Components/ Using Jupyter Notebook to Connect to MRS Spark
Updated on 2024-08-12 GMT+08:00
View PDF
Share

Using Jupyter Notebook to Connect to MRS Spark

Overview

MRS allows you to boost your machine learning, data exploration, and ETL application development efficiency by using PySpark with Jupyter Notebook.

This practice describes how to configure Jupyter Notebook in an MRS cluster to use PySpark.

The detailed steps are as follows:

  1. Step 1: Install the Client on a Node Outside the MRS Cluster
  2. Step 2: Install Python3
  3. Step 3: Install Jupyter Notebook.
  4. Step 4: Verify Jupyter Notebook access to MRS

This practice is available for MRS 3.x and later versions where Python3 is installed on the client node outside the cluster.

Step 1: Install the Client on a Node Outside the MRS Cluster

  1. Prepare a Linux ECS that does not belong to the MRS cluster, associate an EIP to the ECS, and install the cluster client by referring to Installing a Client on a Node Outside the Cluster. For example, the installation directory is /opt/client.
  2. Check whether Kerberos authentication is enabled for the desired MRS cluster.

  3. Log in to FusionInsight Manager of the cluster.
  4. Choose System > Permission > User to create a service user.

    Set User Type to Human-Machine, add hadoop to User Group, select hadoop Primary Group, and add Manager_operator to Role.

    For example, the created user is mrs-test.

    Figure 1 Creating an MRS service user

  5. Log in to the cluster client node as user root and run the following commands to configure environment variables and authenticate the user. Change the user password upon the first user authentication.

    source /opt/client/bigdata_env

    kinit mrs-test

Step 2: Install Python3

  1. Log in to the client node outside the cluster as user root and run the following command to check whether Python3 is installed:

    python3 --version

    • If yes, go to 8.
    • If no, go to 2.

    This case is available only when Python3 is installed on the client node outside the cluster.

  2. Install Python. Python 3.6.6 is used as an example.

    1. Install the following dependencies:

      yum install zlib zlib-devel zip -y

      yum install gcc-c++

      yum install openssl-devel

      yum install sqlite-devel -y

      If the pandas library requires the following dependencies:

      yum install -y xz-devel

      yum install bzip2-devel

    2. Download the source code of the corresponding Python version.

      wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz

    3. Decompress the Python source code package, for example, to the opt directory.

      cd /opt

      tar -xvf Python-3.6.6.tgz

    4. Create a Python installation directory, for example, /opt/python36.

      mkdir /opt/python36

    5. Compile Python.

      cd /opt/python-3.6.6

      ./configure --prefix=/opt/python36

      The following information is displayed if the commands are executed successfully.

      Run the make -j8 command. If the command is successfully executed, the following information is displayed.

      Run the make install command. If the command is successfully executed, the following information is displayed.

    6. Configure Python environment variables.

      export PYTHON_HOME=/opt/python36

      export PATH=$PYTHON_HOME/bin:$PATH

    7. Run the python3 --version command. If the following information is displayed, Python has been installed. 
      Python 3.6.6

  3. Verify Python 3.

    pip3 install helloword

    python3

    import helloworld

    helloworld.say_hello("test")

  4. Install third-party Python libraries (such as pandas and sklearn).

    pip3 install pandas

    pip3 install backports.lzma

    pip3 install sklearn

  5. Run the python3 -m pip list command to check the installation result.

  6. Pack them into Python.zip.

    cd /opt/python36/

    zip -r python36.zip ./*

  7. Upload the file to the specified HDFS directory.

    hdfs dfs -mkdir /user/python

    hdfs dfs -put python36.zip /user/python

  8. Configure the MRS client.

    Go to the Spark client installation directory /opt/client/Spark2x/spark/conf and configure the following parameters in the spark-defaults.conf file:

    spark.pyspark.driver.python=/usr/bin/python3
spark.yarn.dist.archives=hdfs://hacluster/user/python/python36.zip#Python

Step 3: Install Jupyter Notebook.

  1. Log in to the client node as user root and run the following command to install Jupyter Notebook:

    pip3 install jupyter notebook

    The installation is successful if the following command output is displayed.

  2. For security purpose, generate a ciphertext password for logging in to Jupyter and save the password in the Jupyter Notebook configuration file.

    Run the following command and enter the password twice (exit at Out[3]):

    ipython

    [root@ecs-notebook python36]# ipython
Python 3.6.6 (default, Dec 20 2021, 09:32:25)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.2 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from notebook.auth import passwd
In [2]: passwd()
Enter password:
Verify password:
Out[2]: 'argon2:$argon2id$v=19$m=10240,t=10,p=8$g14BqLddl927n/unsyPlLQ$YmoKJzbUfNG7LcxylJzm90bgbKWUIiHy6ZV+ObTzdcA

  3. Generate the Jupyter configuration file.

    jupyter notebook --generate-config

  4. Modify the configuration file.

    vi ~/.jupyter/jupyter_notebook_config.py

    Add the following configurations:

    # -*- coding: utf-8 -*- 
c.NotebookApp.ip='*' #Enter the internal IP address of the ECS.
c.NotebookApp.password = u'argon2:$argon2id$v=19$m=10240,t=10,p=8$NmoAVwd8F6vFP2rX5ZbV7w$SyueJoC0a5TbCuHYzqfSx1vQcFvOTTryR+0uk2MNNZA' # Enter the ciphertext generated at Out[2] in step 2.
c.NotebookApp.open_browser = False # Disable automatic browser opening.
c.NotebookApp.port = 9999 # Specified port number
c.NotebookApp.allow_remote_access = True

Step 4: Verify Jupyter Notebook access to MRS

  1. Run the following command on the client node to start Jupyter Notebook:

    PYSPARK_PYTHON=./Python/bin/python3 PYSPARK_DRIVER_PYTHON=jupyter-notebook PYSPARK_DRIVER_PYTHON_OPTS="--allow-root" pyspark --master yarn --executor-memory 2G --driver-memory 1G

  2. Enter EIP:9999 in the address box of the browser to log in to the Jupyter web UI (ensure that the security group of the ECS allows access from the local public IP address and port 9999). The login password is the one set in 2.

    Figure 2 Logging In to the Jupyter web UI

  3. Create code.

    Create a python3 task and use Spark to read files.

    Figure 3 Creating a Python task

    Log in to FusionInsight Manager and view the submitted PySpark application on the Yarn web UI.

    Figure 4 Viewing the task status

  4. Verify that the pandas library can be called.

    Figure 5 Verifying pandas

FAQs About Interconnection with Jupyter

When the pandas is used for local import, the following error message is displayed.

Perform the following steps to rectify the fault:

  1. Run the python -m pip install backports.lzma command to install the LZMA module.

  2. Go to the /usr/local/python3/lib/python3.6 directory and edit the lzma.py file. The directory varies depending on hosts. You can run the which command to query the directory used by Python.

    Change

    from _lzma import *
from _lzma import _encode_filter_properties, _decode_filter_properties

    To

    try:
    from _lzma import *
    from _lzma import _encode_filter_properties, _decode_filter_properties
except ImportError:
    from backports.lzma import *
    from backports.lzma import _encode_filter_properties, _decode_filter_properties

    Before modification

    After modification

  3. Save the settings and exit, and then run the import command again.