El contenido no se encuentra disponible en el idioma seleccionado. Estamos trabajando continuamente para agregar más idiomas. Gracias por su apoyo.

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
Help Center/ MapReduce Service/ Best Practices/ Interconnection with Ecosystem Components/ Using Jupyter Notebook to Connect to MRS Spark

Using Jupyter Notebook to Connect to MRS Spark

Updated on 2024-08-12 GMT+08:00

Overview

MRS allows you to boost your machine learning, data exploration, and ETL application development efficiency by using PySpark with Jupyter Notebook.

This practice describes how to configure Jupyter Notebook in an MRS cluster to use PySpark.

The detailed steps are as follows:

  1. Step 1: Install the Client on a Node Outside the MRS Cluster
  2. Step 2: Install Python3
  3. Step 3: Install Jupyter Notebook.
  4. Step 4: Verify Jupyter Notebook access to MRS
NOTE:

This practice is available for MRS 3.x and later versions where Python3 is installed on the client node outside the cluster.

Step 1: Install the Client on a Node Outside the MRS Cluster

  1. Prepare a Linux ECS that does not belong to the MRS cluster, associate an EIP to the ECS, and install the cluster client by referring to Installing a Client on a Node Outside the Cluster. For example, the installation directory is /opt/client.
  2. Check whether Kerberos authentication is enabled for the desired MRS cluster.

  3. Log in to FusionInsight Manager of the cluster.
  4. Choose System > Permission > User to create a service user.

    Set User Type to Human-Machine, add hadoop to User Group, select hadoop Primary Group, and add Manager_operator to Role.

    For example, the created user is mrs-test.

    Figure 1 Creating an MRS service user

  5. Log in to the cluster client node as user root and run the following commands to configure environment variables and authenticate the user. Change the user password upon the first user authentication.

    source /opt/client/bigdata_env

    kinit mrs-test

Step 2: Install Python3

  1. Log in to the client node outside the cluster as user root and run the following command to check whether Python3 is installed:

    python3 --version

    • If yes, go to 8.
    • If no, go to 2.
    NOTE:

    This case is available only when Python3 is installed on the client node outside the cluster.

  2. Install Python. Python 3.6.6 is used as an example.

    1. Install the following dependencies:

      yum install zlib zlib-devel zip -y

      yum install gcc-c++

      yum install openssl-devel

      yum install sqlite-devel -y

      If the pandas library requires the following dependencies:

      yum install -y xz-devel

      yum install bzip2-devel

    2. Download the source code of the corresponding Python version.

      wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz

    3. Decompress the Python source code package, for example, to the opt directory.

      cd /opt

      tar -xvf Python-3.6.6.tgz

    4. Create a Python installation directory, for example, /opt/python36.

      mkdir /opt/python36

    5. Compile Python.

      cd /opt/python-3.6.6

      ./configure --prefix=/opt/python36

      The following information is displayed if the commands are executed successfully.

      Run the make -j8 command. If the command is successfully executed, the following information is displayed.

      Run the make install command. If the command is successfully executed, the following information is displayed.

    6. Configure Python environment variables.

      export PYTHON_HOME=/opt/python36

      export PATH=$PYTHON_HOME/bin:$PATH

    7. Run the python3 --version command. If the following information is displayed, Python has been installed.
      Python 3.6.6

  3. Verify Python 3.

    pip3 install helloword

    python3

    import helloworld

    helloworld.say_hello("test")

  4. Install third-party Python libraries (such as pandas and sklearn).

    pip3 install pandas

    pip3 install backports.lzma

    pip3 install sklearn

  5. Run the python3 -m pip list command to check the installation result.

  6. Pack them into Python.zip.

    cd /opt/python36/

    zip -r python36.zip ./*

  7. Upload the file to the specified HDFS directory.

    hdfs dfs -mkdir /user/python

    hdfs dfs -put python36.zip /user/python

  8. Configure the MRS client.

    Go to the Spark client installation directory /opt/client/Spark2x/spark/conf and configure the following parameters in the spark-defaults.conf file:

    spark.pyspark.driver.python=/usr/bin/python3
    spark.yarn.dist.archives=hdfs://hacluster/user/python/python36.zip#Python

Step 3: Install Jupyter Notebook.

  1. Log in to the client node as user root and run the following command to install Jupyter Notebook:

    pip3 install jupyter notebook

    The installation is successful if the following command output is displayed.

  2. For security purpose, generate a ciphertext password for logging in to Jupyter and save the password in the Jupyter Notebook configuration file.

    Run the following command and enter the password twice (exit at Out[3]):

    ipython

    [root@ecs-notebook python36]# ipython
    Python 3.6.6 (default, Dec 20 2021, 09:32:25)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.16.2 -- An enhanced Interactive Python. Type '?' for help.
    In [1]: from notebook.auth import passwd
    In [2]: passwd()
    Enter password:
    Verify password:
    Out[2]: 'argon2:$argon2id$v=19$m=10240,t=10,p=8$g14BqLddl927n/unsyPlLQ$YmoKJzbUfNG7LcxylJzm90bgbKWUIiHy6ZV+ObTzdcA

  3. Generate the Jupyter configuration file.

    jupyter notebook --generate-config

  4. Modify the configuration file.

    vi ~/.jupyter/jupyter_notebook_config.py

    Add the following configurations:

    # -*- coding: utf-8 -*- 
    c.NotebookApp.ip='*' #Enter the internal IP address of the ECS.
    c.NotebookApp.password = u'argon2:$argon2id$v=19$m=10240,t=10,p=8$NmoAVwd8F6vFP2rX5ZbV7w$SyueJoC0a5TbCuHYzqfSx1vQcFvOTTryR+0uk2MNNZA' # Enter the ciphertext generated at Out[2] in step 2.
    c.NotebookApp.open_browser = False # Disable automatic browser opening.
    c.NotebookApp.port = 9999 # Specified port number
    c.NotebookApp.allow_remote_access = True

Step 4: Verify Jupyter Notebook access to MRS

  1. Run the following command on the client node to start Jupyter Notebook:

    PYSPARK_PYTHON=./Python/bin/python3 PYSPARK_DRIVER_PYTHON=jupyter-notebook PYSPARK_DRIVER_PYTHON_OPTS="--allow-root" pyspark --master yarn --executor-memory 2G --driver-memory 1G

  2. Enter EIP:9999 in the address box of the browser to log in to the Jupyter web UI (ensure that the security group of the ECS allows access from the local public IP address and port 9999). The login password is the one set in 2.

    Figure 2 Logging In to the Jupyter web UI

  3. Create code.

    Create a python3 task and use Spark to read files.

    Figure 3 Creating a Python task

    Log in to FusionInsight Manager and view the submitted PySpark application on the Yarn web UI.

    Figure 4 Viewing the task status

  4. Verify that the pandas library can be called.

    Figure 5 Verifying pandas

FAQs About Interconnection with Jupyter

When the pandas is used for local import, the following error message is displayed.

Perform the following steps to rectify the fault:

  1. Run the python -m pip install backports.lzma command to install the LZMA module.

  2. Go to the /usr/local/python3/lib/python3.6 directory and edit the lzma.py file. The directory varies depending on hosts. You can run the which command to query the directory used by Python.

    Change

    from _lzma import *
    from _lzma import _encode_filter_properties, _decode_filter_properties

    To

    try:
        from _lzma import *
        from _lzma import _encode_filter_properties, _decode_filter_properties
    except ImportError:
        from backports.lzma import *
        from backports.lzma import _encode_filter_properties, _decode_filter_properties

    Before modification

    After modification

  3. Save the settings and exit, and then run the import command again.

Utilizamos cookies para mejorar nuestro sitio y tu experiencia. Al continuar navegando en nuestro sitio, tú aceptas nuestra política de cookies. Descubre más

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback