Developing and Scheduling an Import GES Job

Procedure

Assume that the raw data tables in the MySQL database are updated every day. If you want to update the latest graph data generated based on the raw data to GES every day, perform the following steps to compile jobs and periodically schedule the jobs:

1. On the DataArts Studio console, locate a workspace and click DataArts Factory.
2. Create a data development batch processing job and name it import_ges.
   Figure 2 Creating a job
   

3. On the job development page, drag one Dummy node, eight CDM Job nodes, and two Import GES nodes to the canvas, select and drag , as shown in Figure 3.
   The Dummy node serves only as an end identifier. The CDM Job nodes are used to invoke data integration jobs created in Creating a Data Integration Job. The Import GES nodes are used to import graph data to GES.

   Figure 3 Compiling a job
   

4. Configure the eight CDM Job nodes in the job. Invoke the created data integration job to convert the raw data to the standard GES vertex/edge data sets and synchronize the data to OBS and MRS Hive.
   Figure 4 Configuring CDM nodes
   

   CDM node description:

   • vertex_user_rds2hive (CDM Job node): In Node Properties, select the CDM cluster in Creating a Data Integration Job and associate it with the CDM job vertex_user_rds2hive.
   • vertex_user_rds2obs (CDM Job node): In Node Properties, select the CDM cluster in Creating a Data Integration Job and associate it with the CDM job vertex_user_rds2obs.
   • edge_friends_rds2hive (CDM Job node): In Node Properties, select the CDM cluster in Creating a Data Integration Job and associate it with the CDM job edge_friends_rds2hive.
   • edge_friends_rds2obs (CDM Job node): In Node Properties, select the CDM cluster in Creating a Data Integration Job and associate it with the CDM job edge_friends_rds2obs.
   • vertex_movie_rds2hive (CDM Job node): In Node Properties, select the CDM cluster in Creating a Data Integration Job and associate it with the CDM job vertex_movie_rds2hive.
   • vertex_movie_rds2obs (CDM Job node): In Node Properties, select the CDM cluster in Creating a Data Integration Job and associate it with the CDM job vertex_movie_rds2obs.
   • edge_rate_rds2hive (CDM Job node): In Node Properties, select the CDM cluster in Creating a Data Integration Job and associate it with the CDM job edge_rate_rds2hive.
   • edge_rate_rds2obs (CDM Job node): In Node Properties, select the CDM cluster in Creating a Data Integration Job and associate it with the CDM job edge_rate_rds2obs.

5. Configure the two Import GES nodes in the job separately. Only one vertex table and one edge table can be selected for one Import GES node to generate metadata. Therefore, two Import GES nodes are used in this practice to import data in sequence.
   Import GES node description:

   • Import_GES_user-friend: In Node Properties, after a graph name is selected, set the edge data set and vertex data set to the edge_friends edge table and vertex_user vertex table, respectively. In addition, select Overwrite previous repetitive edges. Otherwise, a large number of duplicate edges are generated after periodic scheduling.

     Set Metadata Source to New and click next to Metadata. The New dialog box is displayed, as shown in Figure 6. In the New dialog box, select the edge_friends edge table and vertex_user vertex table in MRS, set Output Directory to the directory where the OBS vertex tables and edge tables are located, and click Create. The system automatically populates the Metadata field to the OBS directory where the generated metadata schema is located.

   • Import_GES_movie-rate: In Node Properties, after a graph name is selected, set the edge data set and vertex data set to the edge_rate edge table and vertex_movie vertex table, respectively. In addition, select Overwrite previous repetitive edges. Otherwise, a large number of duplicate edges are generated after periodic scheduling.

     Set Metadata Source to New and click the generation button next to Metadata. The New dialog box is displayed, as shown in Figure 6. In the New dialog box, select the edge_rate edge table and vertex_movie vertex table in MRS, set Output Directory to the directory where the OBS vertex tables and edge tables are located, and click Create. The system automatically populates the Metadata field to the OBS directory where the generated metadata schema is located.

   Figure 5 Configuring the Import GES node
   
   Figure 6 New
   

6. After configuring the job, click to test it.
   Figure 7 Testing the job
   

7. If the job runs properly, click Scheduling Setup in the right pane and configure the scheduling policy for the job.
   Figure 8 Configuring scheduling type
   

   Parameter descriptions:

   • From: From 00:00 on April 1, 2023, the job is executed at 00:00 every day.
   • Dependency Properties: You can configure a dependency job for this job. You do not need to configure it in this practice.
   • Cross-Cycle Dependency: Select Independent on the previous schedule cycle.

8. Click to save and submit configuration and click to execute the scheduling job. The job will be automatically executed every day, and daily data will be automatically imported to GES.
9. If you want to check the job execution result, choose Monitoring > Monitor Instance in the left navigation pane.
   Figure 9 Viewing the job execution status