On this page

Overview
Step 1: Install the Client on a Node Outside the MRS Cluster
Step 2: Install Python3
Step 3: Install Jupyter Notebook.
Step 4: Verify Jupyter Notebook access to MRS
FAQs About Interconnection with Jupyter

Show all

Help Center/ MapReduce Service/ Best Practices/ Interconnection with Ecosystem Components/ Using Jupyter Notebook to Connect to MRS Spark

Using Jupyter Notebook to Connect to MRS Spark

Updated on 2024-08-12 GMT+08:00

View PDF

Overview

MRS allows you to boost your machine learning, data exploration, and ETL application development efficiency by using PySpark with Jupyter Notebook.

This practice describes how to configure Jupyter Notebook in an MRS cluster to use PySpark.

The detailed steps are as follows:

Step 1: Install the Client on a Node Outside the MRS Cluster
Step 2: Install Python3
Step 3: Install Jupyter Notebook.
Step 4: Verify Jupyter Notebook access to MRS

NOTE:

This practice is available for MRS 3.x and later versions where Python3 is installed on the client node outside the cluster.

Step 1: Install the Client on a Node Outside the MRS Cluster

Prepare a Linux ECS that does not belong to the MRS cluster, associate an EIP to the ECS, and install the cluster client by referring to Installing a Client on a Node Outside the Cluster. For example, the installation directory is /opt/client.
Check whether Kerberos authentication is enabled for the desired MRS cluster.
- If yes, go to 3.
- If no, go to Step 2: Install Python3.
Log in to FusionInsight Manager of the cluster.
Choose System > Permission > User to create a service user.

Set User Type to Human-Machine, add hadoop to User Group, select hadoop Primary Group, and add Manager_operator to Role.

For example, the created user is mrs-test.

Figure 1 Creating an MRS service user
Log in to the cluster client node as user root and run the following commands to configure environment variables and authenticate the user. Change the user password upon the first user authentication.

source /opt/client/bigdata_env

kinit mrs-test

Step 2: Install Python3

Log in to the client node outside the cluster as user root and run the following command to check whether Python3 is installed:

python3 --version
- If yes, go to 8.
- If no, go to 2.
NOTE:

This case is available only when Python3 is installed on the client node outside the cluster.
Install Python. Python 3.6.6 is used as an example.
1. Install the following dependencies:
  yum install zlib zlib-devel zip -y
  
  yum install gcc-c++
  
  yum install openssl-devel
  
  yum install sqlite-devel -y
  
  If the pandas library requires the following dependencies:
  
  yum install -y xz-devel
  
  yum install bzip2-devel
2. Download the source code of the corresponding Python version.
  wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz
3. Decompress the Python source code package, for example, to the opt directory.
  cd /opt
  
  tar -xvf Python-3.6.6.tgz
4. Create a Python installation directory, for example, /opt/python36.
  mkdir /opt/python36
5. Compile Python.
  cd /opt/python-3.6.6
  
  ./configure --prefix=/opt/python36
  
  The following information is displayed if the commands are executed successfully.
  
  Run the make -j8 command. If the command is successfully executed, the following information is displayed.
  
  Run the make install command. If the command is successfully executed, the following information is displayed.
6. Configure Python environment variables.
  export PYTHON_HOME=/opt/python36
  
  export PATH=$PYTHON_HOME/bin:$PATH
7. Run the python3 --version command. If the following information is displayed, Python has been installed.
```
Python 3.6.6
```
Verify Python 3.

pip3 install helloword

python3

import helloworld

helloworld.say_hello("test")
Install third-party Python libraries (such as pandas and sklearn).

pip3 install pandas

pip3 install backports.lzma

pip3 install sklearn
Run the python3 -m pip list command to check the installation result.
Pack them into Python.zip.

cd /opt/python36/

zip -r python36.zip ./*
Upload the file to the specified HDFS directory.

hdfs dfs -mkdir /user/python

hdfs dfs -put python36.zip /user/python
Configure the MRS client.

Go to the Spark client installation directory /opt/client/Spark2x/spark/conf and configure the following parameters in the spark-defaults.conf file:
```
spark.pyspark.driver.python=/usr/bin/python3
spark.yarn.dist.archives=hdfs://hacluster/user/python/python36.zip#Python
```

Step 3: Install Jupyter Notebook.

Log in to the client node as user root and run the following command to install Jupyter Notebook:

pip3 install jupyter notebook

The installation is successful if the following command output is displayed.

For security purpose, generate a ciphertext password for logging in to Jupyter and save the password in the Jupyter Notebook configuration file.

Run the following command and enter the password twice (exit at Out[3]):

ipython

[root@ecs-notebook python36]# ipython
Python 3.6.6 (default, Dec 20 2021, 09:32:25)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.2 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from notebook.auth import passwd
In [2]: passwd()
Enter password:
Verify password:
Out[2]: 'argon2:$argon2id$v=19$m=10240,t=10,p=8$g14BqLddl927n/unsyPlLQ$YmoKJzbUfNG7LcxylJzm90bgbKWUIiHy6ZV+ObTzdcA

Generate the Jupyter configuration file.

jupyter notebook --generate-config

Modify the configuration file.

vi ~/.jupyter/jupyter_notebook_config.py

Add the following configurations:

# -*- coding: utf-8 -*- 
c.NotebookApp.ip='*' #Enter the internal IP address of the ECS.
c.NotebookApp.password = u'argon2:$argon2id$v=19$m=10240,t=10,p=8$NmoAVwd8F6vFP2rX5ZbV7w$SyueJoC0a5TbCuHYzqfSx1vQcFvOTTryR+0uk2MNNZA' # Enter the ciphertext generated at Out[2] in step 2.
c.NotebookApp.open_browser = False # Disable automatic browser opening.
c.NotebookApp.port = 9999 # Specified port number
c.NotebookApp.allow_remote_access = True

Step 4: Verify Jupyter Notebook access to MRS

Run the following command on the client node to start Jupyter Notebook:

PYSPARK_PYTHON=./Python/bin/python3 PYSPARK_DRIVER_PYTHON=jupyter-notebook PYSPARK_DRIVER_PYTHON_OPTS="--allow-root" pyspark --master yarn --executor-memory 2G --driver-memory 1G
Enter EIP:9999 in the address box of the browser to log in to the Jupyter web UI (ensure that the security group of the ECS allows access from the local public IP address and port 9999). The login password is the one set in 2.

Figure 2 Logging In to the Jupyter web UI
Create code.

Create a python3 task and use Spark to read files.

Figure 3 Creating a Python task

Log in to FusionInsight Manager and view the submitted PySpark application on the Yarn web UI.

Figure 4 Viewing the task status
Verify that the pandas library can be called.

Figure 5 Verifying pandas

FAQs About Interconnection with Jupyter

When the pandas is used for local import, the following error message is displayed.

Click to enlarge

Perform the following steps to rectify the fault:

Run the python -m pip install backports.lzma command to install the LZMA module.

Go to the /usr/local/python3/lib/python3.6 directory and edit the lzma.py file. The directory varies depending on hosts. You can run the which command to query the directory used by Python.

Change

from _lzma import *
from _lzma import _encode_filter_properties, _decode_filter_properties

try:
    from _lzma import *
    from _lzma import _encode_filter_properties, _decode_filter_properties
except ImportError:
    from backports.lzma import *
    from backports.lzma import _encode_filter_properties, _decode_filter_properties

Before modification

Click to enlarge

After modification

Click to enlarge

Save the settings and exit, and then run the import command again.

Parent topic: Interconnection with Ecosystem Components

Previous topic: Interconnecting MRS Kafka with Kafka Eagle

Next topic: MRS Cluster Management

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot