Developing Tasks

Prerequisites

Notebooks are enabled, and a notebook instance has been created. For how to create a notebook instance, see Creating a Notebook Instance.

Notes and Constraints

• Only the DLI data source is supported.

Developing a Notebook

1. In the navigation pane on the DataArts Factory console, choose Data Development > Notebook.
2. Click Open to go to the notebook development page.
3. Click and select Notebook to create a notebook file. Add an .ipynb file to create a notebook. Then develop a notebook.

   

4. On the notebook development page, enter and debug development code.
   NOTE:

   • You can save, insert data below, cut, copy, paste, run, and interrupt data.
   • You can click in front of a line to run the code.
   • Three types of code display styles are supported: Code, Markdown, and Raw.
   • You can insert data below, move up, move down, and delete code lines.

5. Click to run the code.
6. View the script execution result.

Developing a Launcher

1. Click and select Launcher to access the launcher development page. You can develop DLI and Python tasks.
   

   • Develop a DLI task.
     NOTE:

     Before developing a DLI task, create an agency for DLI and authorize the agency. For example, you can create an agency named notebook_agency and assign DLI FullAccess and OBS Administrator to the agency. For details, see Creating a Custom DLI Agency.

     Figure 1 Creating and authorizing an agency
     

     In addition, configure the following custom policy in Permissions > Policies/Roles.

     

     {
         "Version": "1.1",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "iam:agencies:pass"
                 ]
             }
         ]
     }
     1. Click DLI-PySpark-3.1.1 to go to the DLI development page.
     2. In the upper right corner of the page, click connect to configure connection parameters.
        • Configure basic parameters Pool and Queue (indicating the DLI resource pool and queue name).
        • Configure advanced settings.

          Table 1 Spark Config

          Parameter	Description
          --conf	Parameters for running the DLI task, such as the name of the DLI task agency spark.dli.job.agency.name=notebook_agency (mandatory) Set other parameters as required.
          --jars	OBS path of the file. Separate multiple paths by pressing Enter. (optional)
          --py-files	OBS path of the file. Separate multiple paths by pressing Enter. (optional)
          --files	OBS path of the file. Separate multiple paths by pressing Enter. (optional)

          Table 2 Resource Config

          Parameter	Description
          Driver Memory	Driver memory, which ranges from 0 (excluded) to 16. The default value is 1.
          Driver Cores	Number of driver cores, which ranges from 0 (excluded) to 4. The default value is 1.
          Executor Memory	Executor memory, which ranges from 0 (excluded) to 16. The default value is 1.
          Executor Cores	Number of executor cores, which ranges from 0 (excluded) to 4. The default value is 1.
          Executors	Number of executors, which ranges from 0 (excluded) to 16. The default value is 1.

        • Click Connect. After the configuration is complete, the DLI queue information and cluster status "cluster status: connected" are displayed.
     3. Click the code line, enter the development code, and debug the code.
     4. Click in front of the code line to run the code.

        Table 3

        Example Code	Execution Result
        %%spark spark.read.parquet('obs://mytestbucket/demo/data.parquet').show()
        %%sql show tables
        %scala import org.apache.spark.sql.SparkSession val spark = SparkSession.builder().appName("demo").getOrCreate(); val inputFile = "obs://mytestbucket/demo/test.txt" val outputDir = "obs://mytestbucket/demo/test" val textFile = spark.read.textFile(inputFile) val wordCounts = textFile.flatMap(line => line.split(" ")).groupByKey(identity).count() wordCounts.write.format("csv").save(outputDir) wordCounts.show() spark.stop()
        %python from pyspark.sql import SparkSession spark = SparkSession.builder.appName("PySpark Example").getOrCreate() data = [("Alice", 34), ("Bob", 45), ("Charlie", 29), ("David", 35)] columns = ["Name", "Age"] df = spark.createDataFrame(data, columns) df.show() filtered_df = df.filter(df.Age > 30) filtered_df.show() average_age = df.groupBy().avg("Age").collect()[0][0] print(f"Average Age: {average_age}") spark.stop()

   • Develop a Python task.
     1. Click python-3.7.10 to access the Python development page.
     2. Click the code line, enter the development code, and debug the code.
     3. Click in front of the code line to run the code.

2. Save and run the code.
3. View the code execution result.

More Notebook Operations

For details about more notebook operations, see Common Functions of JupyterLab.

Common Cell Operations