Operating Environment and Data Preparation

Preparing the Environment

• If you are new to DataArts Studio, register a Huawei account, buy a DataArts Studio instance, create workspaces, and make other preparations. For details, see Buying and Configuring a DataArts Studio Instance. Then you can go to the created workspace and start using DataArts Studio.

• Create an MRS cluster that contains the Hive component on the MRS console. Metadata can be generated from the vertex and edge data sets in the cluster. When creating the MRS cluster, ensure that its network parameter settings (including the region, VPC, subnet, and security group) are consistent with those of the CDM cluster in the DataArts Studio instance so that the MRS cluster can communicate with the CDM cluster through the internal network. Otherwise, you need to manually enable the communication between the MRS cluster and the CDM cluster. In addition, ensure that the two clusters use the same enterprise project.
  

  During the creation of an MRS cluster, a security group is automatically created. You are advised to create an MRS security cluster first and then buy a DataArts Studio instance, selecting the same VPC and subnet as the MRS cluster and the security group (named in mrs_Cluster name_Random character format) that is automatically created. This ensures that the DataArts Studio instance can communicate with the MRS cluster by default.

  If you already have a DataArts Studio instance before creating an MRS cluster, you need to choose Access Control > Security Groups on the VPC console and add a rule to allow inbound traffic to the security group (named in mrs_Cluster name_Random character format) created by the MRS cluster. For details, see Configuring Security Group Rules.

• Create a MySQL DB instance on the RDS console to simulate the data source. When creating the MySQL DB instance, ensure that its network parameter settings (including the region, VPC, subnet, and security group) are consistent with those of the CDM cluster in the DataArts Studio instance so that the MySQL DB instance can communicate with the CDM cluster through the internal network. Otherwise, you need to manually enable the communication between the MySQL DB instance and the CDM cluster. In addition, ensure that the MySQL DB instance and the CDM cluster use the same enterprise project.
• Prepare an OBS bucket to store the generated metadata. The OBS bucket must be in the same region as the CDM cluster in the DataArts Studio instance, and the enterprise project of the OBS bucket must be the same as that of the CDM cluster.
• Create a graph on the GES console to import graph data for visualized graph analysis. GES must be in the same region as the CDM cluster in the DataArts Studio instance, and the enterprise project of GES must be the same as that of the CDM cluster.

Preparing Data Sources

In this practice, the raw data includes the user table vertex_user, movie table vertex_movie, friend relationship table edge_friends, and movie rating table edge_rate. Figure 1 shows the relationships between them.

Figure 1 Graph data description

To facilitate demonstration, this practice provides some data used to simulate the original data. To integrate the source data into the cloud, you need to store the sample data in CSV files and upload them to an OBS bucket.

1. Create CSV files (UTF-8 without BOM), name the files with the corresponding data table names, copy the sample data to different CSV files, and save the files.
   To generate a CSV file in Windows, you can perform the following steps:

   1. Use a text editor (for example, Notepad) to create a .txt document and copy the sample data to the document. Then check the total number of rows and check whether the data of rows is correctly separated. (If the sample data is copied from a PDF document, the data in a single row will be wrapped if the data is too long. In this case, you must manually adjust the data to ensure that it is in a single row.)
   2. Choose File > Save as. In the displayed dialog box, set Save as type to All files (*.*), enter the file name with the .csv suffix for File name, and select the UTF-8 encoding format (without BOM) to save the file in CSV format.

2. Upload the CSV file to OBS.
   1. Log in to the management console and choose Storage > Object Storage Service to access the OBS console.
   2. Click Create Bucket and set parameters as prompted to create an OBS bucket named fast-demo.
      

      To ensure network connectivity, select the same region for OBS bucket as that for the DataArts Studio instance. If an enterprise project is required, select the enterprise project that is the same as that of the DataArts Studio instance.

      For details about how to create a bucket on the OBS console, see Creating a Bucket in Object Storage Service Console Operation Guide.

   3. Upload data to OBS bucket fast-demo.

      For details about how to upload a file on the OBS console, see Uploading a File in Object Storage Service Console Operation Guide.

Creating a Data Connection in Management Center

In this practice, you need to synchronize data from the MySQL database to MRS Hive, standardize the data based on the GES graph import requirements, and generate metadata using MRS Hive.

Therefore, you need to create an MRS connection in Management Center. The procedure is as follows:

1. Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
2. On the DataArts Studio console, locate a workspace and click Management Center.
3. On the displayed Manage Data Connections page, click Create Data Connection.
   Figure 2 Creating a data connection
   

4. In the dialog box displayed, set data connection parameters and click OK.

   The following part describes how to create an MRS Hive connection. See Figure 3 for details.

   • Data Connection Type: MRS Hive is selected by default.
   • Name: Enter mrs_hive_link.
   • Tag: Enter a new tag name or select an existing tag from the drop-down list box. This parameter is optional.
   • Applicable Modules: Retain the default settings.
   • Connection Type: Select Proxy connection.
   • Manual: Select Cluster Name Mode. IP and Port are automatically set.
   • MRS Cluster Name: Select an existing MRS cluster.
   • KMS Key: Select a KMS key and use it to encrypt sensitive data. If no KMS key is available, click Access KMS to go to the KMS console and create one.
   • Agent: Select a DataArts Migration cluster as the connection agent. The DataArts Migration cluster and MRS cluster must be in the same region, AZ, VPC, and subnet, and the security group rule must allow communication between the two clusters. In this example, select the DataArts Migration cluster that is automatically created during DataArts Studio instance creation.

     To connect to an MRS 2.x cluster, select the DataArts Migration cluster of the 2.x version as the agent.

   • Username: Enter the Kerberos authentication user. In an MRS policy, user admin is the default management user and cannot be used as the authentication user of the cluster that uses Kerberos authentication. Therefore, to create a connection for an MRS cluster that uses Kerberos authentication, perform the following operations:
     1. Log in to MRS Manager as user admin.
     2. Choose System > Permission > Security Policy > Password Policy. Click Add Password Policy and add a policy under which the password never expires.
        • Set Password Policy Name to neverexp.
        • Set Password Validity Period (Days) to 0, indicating that the password never expires.
        • Set Password Expiration Notification (Days) to 0.
        • Retain the default values for other parameters.
     3. Choose System > Permission > User. On the page displayed, click Create to add a dedicated user as the Kerberos authentication user and set the password policy to neverexp. Select the user group superGroup for the user, and assign all roles to the user.
        

        • For clusters of MRS 3.1.0 or later, the user must at least have permissions of the Manager_viewer role to create data connections in Management Center. To perform database, table, and data operations on components, the user must also have user group permissions of the components.
        • For clusters earlier than MRS 3.1.0, the user must have permissions of the Manager_administrator or System_administrator role to create data connections in Management Center.
        • A user with only the Manager_tenant or Manager_auditor permission cannot create connections.
     4. Log in to Manager as the new user and change the initial password. Otherwise, the connection fails to be created.
     5. Synchronize IAM users.
        1. Log in to the MRS console.
        2. Choose Clusters > Active Clusters, select a running cluster, and click its name to go to its details page.
        3. In the Basic Information area of the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.
           

           • When the policy of the user group to which the IAM user belongs changes from MRS ReadOnlyAccess to MRS CommonOperations, MRS FullAccess, or MRS Administrator, wait for 5 minutes until the new policy takes effect after the synchronization is complete because the SSSD (System Security Services Daemon) cache of cluster nodes needs time to be updated. Then, submit a job. Otherwise, the job may fail to be submitted.
           • When the policy of the user group to which the IAM user belongs changes from MRS CommonOperations, MRS FullAccess, or MRS Administrator to MRS ReadOnlyAccess, wait for 5 minutes until the new policy takes effect after the synchronization is complete because the SSSD cache of cluster nodes needs time to be updated.
   • Password: Enter the password of the Kerberos authentication user.
   Figure 3 Creating an MRS Hive data connection
   

Creating Data Tables

To facilitate demonstration, you need to import the sample data in CSV format to the MySQL database using DataArts Migration. Then, the MySQL database functions as the data source. You need to create raw data tables in the MySQL database before importing data.

In the formal service process, the source data of the MySQL database needs to be imported to the OBS database as the vertex and edge data sets. In that case, you do not need to create tables in advance. However, before importing source data from the MySQL database to MRS Hive, you need to create standard data tables in the MRS Hive database in advance.

Therefore, in this practice, you need to create raw data tables in the MySQL database and standard data tables in the MRS Hive database. This section describes how to create tables using SQL statements.

1. Create a raw data table in the MySQL database. Run the following SQL statements in the MySQL database to create four raw data tables based on the raw data structure in Preparing Data Sources:

   DROP TABLE IF EXISTS `edge_friends`;
   CREATE TABLE `edge_friends` (
       `user1` varchar(32) DEFAULT NULL,
       `user2` varchar(32) DEFAULT NULL
   );
   
   DROP TABLE IF EXISTS `edge_rate`;
   CREATE TABLE `edge_rate` (
       `user` varchar(32) DEFAULT NULL,
       `movie` varchar(64) DEFAULT NULL,
       `score` int(11) unsigned DEFAULT NULL,
       `datatime` varchar(32) DEFAULT NULL
   );
   
   DROP TABLE IF EXISTS `vertex_movie`;
   CREATE TABLE `vertex_movie` (
       `movie` varchar(64) DEFAULT NULL,
       `year` varchar(32) DEFAULT NULL,
       `genres` varchar(64) DEFAULT NULL
   );
   
   DROP TABLE IF EXISTS `vertex_user`;
   CREATE TABLE `vertex_user` (
       `user` varchar(32) DEFAULT NULL,
       `gender` varchar(32) DEFAULT NULL,
       `age` varchar(32) DEFAULT NULL,
       `occupation` varchar(32) DEFAULT NULL,
       `zip-code` varchar(32) DEFAULT NULL
   );

2. Create standard data tables in the MRS Hive database.

   Standardize the raw data structure based on the GES graph import requirements, that is, add labels to the second columns of vertex tables vertex_user and vertex_movie, and add labels to the third columns of edge tables edge_rate and edge_friends.

   

   The vertex and edge data sets must comply with the data format requirements of GES graphs. The graph data format requirements are briefed as follows. For details, see Graph Data Formats.

   • The vertex data set contains the data of each vertex. Each row is the data of a vertex. The format is as follows. id is the unique identifier of vertex data.

     id,label,property 1,property 2,property 3,...

   • The edge data set contains the data of each edge. Each row is the data of an edge. Graph specifications in GES are defined based on the edge quantity, for example, one million edges. The format is as follows. id 1 and id 2 are the IDs of the two endpoints of an edge.

     id 1, id 2, label, property 1, property 2,...

   On the DataArts Factory console, you can select the MRS Hive connection created in Creating a Data Connection in Management Center, select a database, and run the following SQL statement to create a standard data table in the MRS Hive database.

   Figure 4 Creating a standard data table in the MRS Hive database
   

   DROP TABLE IF EXISTS `edge_friends`;
   CREATE TABLE test_ges.`edge_friends` (
       `user1` STRING COMMENT '',
       `user2` STRING COMMENT '',
       `label` STRING COMMENT ''
   );
   
   DROP TABLE IF EXISTS `edge_rate`;
   CREATE TABLE test_ges.`edge_rate` (
       `user` STRING COMMENT '',
       `movie` STRING COMMENT '',
       `label` STRING COMMENT '',
       `score` INT COMMENT '',
       `datatime` STRING COMMENT ''
   );
   
   DROP TABLE IF EXISTS `vertex_movie`;
   CREATE TABLE test_ges.`vertex_movie` (
       `movie` STRING COMMENT '',
       `label` STRING COMMENT '',
       `year` STRING COMMENT '',
       `genres` STRING COMMENT ''
   );
   
   DROP TABLE IF EXISTS `vertex_user`;
   CREATE TABLE test_ges.`vertex_user` (
       `user` STRING COMMENT '',
       `label` STRING COMMENT '',
       `gender` STRING COMMENT '',
       `age` STRING COMMENT '',
       `occupation` STRING COMMENT '',
       `zip-code` STRING COMMENT ''
   );