Using a Notebook Instance for Online Development and Debugging
This section describes how to develop tasks using a notebook instance.
Prerequisites
There is a running notebook instance. For how to create a notebook instance, see Creating a Runtime Environment.
Constraints
- You can create a maximum of 10 notebooks in the current workspace.
- Only MRS Spark and Fabric SQL data types are supported.
- You can only use the notebook instances you created to develop notebook jobs.
- You can bind compute resources only if you are the workspace administrator, have the DAYU Administrator or Tenant Administrator permissions, or have been assigned a custom role with the create permission.
- If you are assigned both a preset role and a custom role, you have permissions of both roles.
- To bind the MRS Spark compute engine, you must create an MRS Spark cluster of version 3.3.1 or later first.
- When notebook jobs are interconnected with MRS Spark, individual-based fine-grained data permission control is not supported.
Binding Compute Resources
If you have bound a compute resource to a notebook, you can click the default engine Python (which does not depend on compute resources) in the upper right corner of the code development area, and select the compute resource from the drop-down list.
If no compute resource has been bound to the notebook, perform the following steps to bind a compute resource. You can also bind a compute resource on the Management Center page by referring to Binding or Unbinding a Compute Resource.
- Log in to the DataArts Studio console.
For details, see Accessing the DataArts Studio Instance Console.
- On the DataArts Studio console, locate a workspace and click DataArts Factory.
- In the project directory or notebook environment directory, double-click the notebook file you want to edit.
- On the notebook development page, click the default engine Python (which does not depend on compute resources) or another compute engine in the upper right corner.
- Select Bind Compute Resource from the drop-down list box.
Table 1 Binding a compute resource Parameter
Description
Resource Type
- MRS Spark
- Fabric SQL
Resource Name
Name of the resource
MRS Cluster
This parameter is mandatory when MRS Spark is selected for Resource Type.
Select an MRS cluster. For details about how to buy an MRS cluster, see Buying MRS Clusters. After the MRS cluster is created, you can view the basic information about the MRS cluster in Active Clusters.
Workspace
This parameter is mandatory when Fabric SQL is selected for Resource Type.
A workspace is the basic unit of Fabric. Select a workspace for Fabric SQL.
You can create a resource workspace on the DataArts Fabric console. For details about how to create a workspace, see Creating a Workspace.
Endpoint
This parameter is mandatory when Fabric SQL is selected for Resource Type.
It specifies the compute resources required for executing SQL tasks.
Select an endpoint after selecting a compute resource workspace. For details about how to create a SQL endpoint, see Creating a SQL Endpoint.
- Click OK to bind the compute resource.
You can view the bound compute resource by referring to Managing Notebook Resources. You can also bind compute resources by referring to Managing Notebook Resources.
Viewing Dependency Packages
During development, you can click Dependencies in the right sidebar to view the dependencies in the current runtime environment.
If there is no package in the environment, you need to install required packages using statements.
Developing and Debugging a Notebook
- Go to the data development page and double-click the notebook file you want to edit in the project directory or notebook environment directory.
- On the notebook development page, enter and debug development code. You can select a desired compute engine in the upper right corner.
- By default, the Python engine is used, which does not depend on compute resources.
- You can also choose the Fabric SQL or MRS Spark compute engine. Click the default Python engine to view information about the created compute engines. You can choose the compute engine you need. A notebook file supports only one compute engine. After you select a compute engine, all cells in the notebook file use this compute engine.
- You need to bind compute resources to compute engines. For details about how to bind compute resources, see Binding Compute Resources.
- The following magic commands are supported: %mrs_spark, %%spark, and %aura_frame.
- Optional: Enter content in Markdown format. Notebook supports basic Markdown syntax, KaTeX formulas, and Mermaid chart rendering.
- Click New Cell to create a cell.
- In the new cell, click Python in the upper right corner and select Markdown.
- Enter content in the cell. Click Show preview in the upper left corner. You can preview the content on the right in real time.
Click
in the upper right corner of the cell or click the blank area outside the cell. The cell automatically switches to the overview mode. You can then double-click the cell to switch to the editing mode.
- Click Save to save the content.
- Click
and select Run cell. The code of the entire cell will be executed. After the execution is complete, if a message is displayed indicating that the execution is successful, you can view the execution result. If the code execution fails, you can view the possible failure cause in the execution result. %spark info
Figure 1 Execution result example 1
%%spark spark.sql("SHOW TABLES").show()Figure 2 Execution result example 2
a = "123111" * 2 print(a) a = "12311" * 3 print(a) a = "123eee" * 5 print(a) a = "123aa1122" * 2 print(a) a = "123absg3wut235456&&&" * 2 print(a)
Figure 3 Execution result example 3
Figure 4 Execution result example 4
%%spark config { "driverMemory": "2G", "driverCores": 1, "executorMemory": "2G", "executorCores": 1, "conf": { "spark.pyspark.python": "./temp_env/bin/python", "spark.yarn.dist.archives":"obs://notebook-dataarts/python_env/pyspark_conda_env.tar.gz#temp_env", "spark.jars": "obs://notebook-dataarts/mrs_jars/spark-utils-1.0-SNAPSHOT.jar" } }Figure 5 Execution result example 5
%%spark import pandas print(pandas.__version__)
Create a package that contains pandas so that pandas can be referenced.
Figure 6 Execution result example 6
%mrs_spark
Figure 7 Execution result example 7
%%spark import my_script print(my_script.add_numbers(10, 20)) rdd = sc.parallelize(["apple", "banana", "cherry"]) result = rdd.map(my_script.format_string).collect() for item in result: print(item)Figure 8 Execution result example 8
- Python is displayed in the upper right corner of each development cell. You can click Python and select Markdown from the drop-down list to switch the cell type.
- The following operations are supported: Run All (running all cells), Save (saving all cells), Submit to Project Directory, New Cell (creating a cell), Clear Outputs (clearing all cells), and More (Restart Kernel and Kill Kernel). These operations apply to the entire notebook file.
- You can copy, paste, and cut cells, add a cell above or below the current cell, move a cell up or down, and clear execution results.
- You can debug part of the code not in a Markdown cell. Select part of the code, click
, and select Run selected text to run the selected code. You can also run code in the following ways: - Select Run all above to run all non-Markdown cells above the current cell.
- Select Run all below to run the current cell and all non-Markdown cells below it.
: Delete a cell.
: Maximize the cell window. After the window is maximized, only the current cell is displayed.
: Minimize the cell window. After the window is minimized, you can view all cells.
: After maximizing the window, click
to view the previous cell.
: After maximizing the window, click
to view the next cell.
Submitting a Notebook from an Environment Directory to a Project Directory
After creating a notebook, you need to submit it from the environment directory to a project directory.
- Go to the notebook development page.
- Click Submit to Project Directory on the top of the cell.
- Set Name and Path of the notebook job. Figure 9 Submit to Project Directory
- The name can contain only letters, digits, hyphens (-), underscores (_), and periods (.).
- The name can be customized.
- The path is used to store the notebook job.
- Click OK.
- View the submitted notebook job in the job directory tree.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot