هذه الصفحة غير متوفرة حاليًا بلغتك المحلية. نحن نعمل جاهدين على إضافة المزيد من اللغات. شاكرين تفهمك ودعمك المستمر لنا.

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive

Developing an SQL Script

Updated on 2022-08-17 GMT+08:00

You can develop, debug, and run SQL scripts online. The developed scripts can be run in jobs. For details, see Developing a Job.

Prerequisites

  • A corresponding cloud service has been enabled and a database has been created in the cloud service. The Flink SQL script does not involve this operation.
  • A data connection that matches the data connection type of the created script. For details, see Creating Data Connections. The Flink SQL script does not involve this operation.
  • You have locked the script. Otherwise, you must click Lock so that you can develop the script. A script you create or import is locked by you by default. For details, see the lock function.

Procedure

  1. Log in to the DataArts Studio console. Locate an instance and click Access. On the displayed page, locate a workspace and click DataArts Factory.
    Figure 1 DataArts Factory
  2. In the left navigation pane of DataArts Factory, choose Data Development > Develop Script.
  3. In the script directory, double-click a script to access the script development page.
  4. In the upper part of the editor, select script properties. Table 1 describes the script properties. Skip this step when creating a Flink SQL script.
    Table 1 SQL script properties

    Property

    Description

    Data Connection

    Selects a data connection.

    Database

    Name of the database.

    Resource Queue

    Selects a resource queue for executing a DLI job. Set this parameter when a DLI or SQL script is created.

    You can create a resource queue using either of the following methods:
    • Click . The Queue Management page of DLI is displayed.
    • Go to the DLI console.
    NOTE:

    DLI provides the default resource queue default, which does not support insert, load, or cat commands.

    To set properties for submitting SQL jobs in the form of key/value, click . A maximum of 10 properties can be set. The properties are described as follows:
    • dli.sql.autoBroadcastJoinThreshold: specifies the data volume threshold to use BroadcastJoin. If the data volume exceeds the threshold, BroadcastJoin will be automatically enabled.
    • dli.sql.shuffle.partitions: specifies the number of partitions during shuffling.
    • dli.sql.cbo.enabled: specifies whether to enable the CBO optimization policy.
    • dli.sql.cbo.joinReorder.enabled: specifies whether join reordering is allowed when CBO optimization is enabled.
    • dli.sql.multiLevelDir.enabled: specifies whether to query the content in subdirectories if there are subdirectories in the specified directory of an OBS table or in the partition directory of an OBS partition table. By default, the content in subdirectories is not queried.
    • dli.sql.dynamicPartitionOverwrite.enabled: specifies that only partitions used during data query are overwritten and other partitions are not deleted.
  5. Enter an SQL statement in the editor. You can enter multiple SQL statements.
    NOTE:
    • Note that the system date obtained by using an SQL statement is different from that obtained by using the database tool. The query result is stored in the database in the YYYY-MM-DD format, but the query result displayed on the page is in the converted format.
    • SQL statements are separated by semicolons (;). If semicolons are used in other places but not used to separate SQL statements, escape them with backslashes (\). For example:
      select 1;
      select * from a where b="dsfa\;";  --example 1\;example 2.
    To facilitate script development, DataArts Factory provides the following capabilities:
    • The script editor supports the following shortcut keys, which improve the script development efficiency:
      • Ctrl + /: Comment out or uncomment the line or code block at the cursor.
      • Ctrl + S: Save
      • Ctrl +Z: Cancel
      • Ctrl + Y: Redo
      • Ctrl + F: Search
      • Ctrl + Shift + R: Replace
      • Ctrl + X: Cut (cut a line when the cursor selects nothing).
      • Alt + mouse dragging: Select columns to edit a block.
      • Ctrl + mouse click: Select multiple lines to edit or indent them together.
      • Shift + Ctrl + K: Delete the current line.
      • Ctrl + (or ): Move the cursor rightwards (or leftwards) by word.
      • Ctrl + Home or Ctrl + End: Navigate to the beginning or end of the current file.
      • Home or End: Navigate to the beginning or end of the current line.
      • Ctrl + Shift + L: Double-click all the same character strings and add cursors to them to implement batch modification.
    • System functions (Flink SQL, Spark SQL, ClickHouse SQL, and Presto SQL do not support system functions.)

      To view the functions supported by this type of data connection, click System Function on the right of the editor. You can double-click a function to the editor to use it.

    • Data tables can be read to generate SQL statements. (Flink SQL, Spark SQL, ClickHouse SQL, and Presto SQL do not support this function.)

      Click Data Tables on the right of the editor to display all the tables in the current database or schema. You can select tables and columns and click Generate SQL Statement in the lower right corner to generate an SQL statement, which you need to manually format.

    • Script parameters (Currently, only Flink SQL does not support script parameters.)

      You can directly write script parameters in SQL statements. When debugging scripts, you can enter parameter values in the script editor. If the script is referenced by a job, you can set parameter values on the job development page. The parameter values can use EL expressions (see Expression Overview).

      In the following script example, str1 indicates the parameter name. It can contain only letters, numbers, hyphens (-), underscores (_), greater-than signs (>), and less-than signs (<), and can contain a maximum of 16 characters. The parameter name must be unique.

      select ${str1} from data;
      For MRS Spark SQL and MRS Hive SQL scripts, you set a program parameter by referring to set hive.exec.parallel=true; in the SQL statements or configure this parameter by setting Program Parameter on Node Properties of the job.
      Figure 2 Program Parameter
    • Owner

      Click Basic Info to set the script owner and description.

  6. (Optional) In the upper part of the editor, click Format to format the SQL statement. When developing a Flink SQL script, skip this step.
  7. In the upper part of the editor, click Execute. If you need to execute some SQL statements separately, select the SQL statements first. After executing the SQL statement, view the execution history and result of the script in the lower part of the editor. When developing a Flink SQL script, skip this step.
    NOTE:
    • You can perform the following operations on execution results:
      • Double-click or right-click the name of an execution result tab to rename it. The name can contain a maximum of 16 characters.
      • Right-click the name of an execution result tab to close the current tab, all the tabs to the left or right of the current tab, all the other tabs, or all the tabs.
    • If the MRS cluster is a non-security cluster and the command whitelist is not restricted, you can easily find the corresponding task on the Yarn management page of MRS based on the script name and execution time after adding the application name information during Hive SQL execution. Note that if the default engine is tez, you need to set the engine to mr to disable the tez engine.
  8. Above the editor, click to save the script.
    If the script is created but not saved, set the parameters listed in Table 2.
    Table 2 Script parameters

    Parameter

    Mandatory

    Description

    Script Name

    Yes

    Name of the script. The name contains a maximum of 128 characters, including only letters, numbers, hyphens (-), underscores (_), and periods (.).

    Owners

    No

    Owner of the script. By default, the creator of the script is the owner.

    Description

    No

    Descriptive information about the script.

    Select Directory

    Yes

    Directory to which the script belongs. The root directory is selected by default.

    NOTE:

    If you open an unsaved script, you can restore its content from the local cache.

Downloading or Dumping a Script Execution Result

Constraints: This function is available only when the OBS service is available.

After the script is executed successfully, you can download or dump the execution result. Only users with the DAYU Administrator or Tenant Administrator policy can download or dump execution results..

  • Download result: Download the CSV result files to the local host.
  • Dump result: Dump the CSV result files to OBS. For details, see Table 3.
    NOTE:

    The execution results of Flink SQL scripts, RDS SQL scripts, and shell scripts cannot be dumped.

    Table 3 Parameters for dumping results

    Parameter

    Mandatory

    Description

    Data Format

    Yes

    Format of the data to be exported. Only CSV result files can be exported.

    Resource Queue

    No

    DLI queue where the export operation is to be performed. Set this parameter when a DLI or SQL script is created.

    Compression Format

    No

    Format of compression. Set this parameter when a DLI or SQL script is created.

    • none
    • bzip2
    • deflate
    • gzip

    Storage Path

    Yes

    OBS path where the result file is stored. After selecting an OBS path, customize a folder. Then, the system will create it automatically for storing the result file.

    Cover Type

    No

    If a folder that has the same name as your custom folder exists in the storage path, select a cover type. Set this parameter when a DLI or SQL script is created.

    • Overwrite: The existing folder will be overwritten by the customized folder.
    • Report: The system reports an error and suspends the export operation.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback