|
Making preparations |
The preparations include preparing data lakes and preparing and synchronizing data.
Preparing data lakes:
In enterprise mode, the development environment and production environment need to be isolated. Therefore, you need to prepare two data lake services, one for the production environment and the other for the development environment.
- For clustered data sources, such as MRS, GaussDB(DWS), RDS, MySQL, Oracle, DIS, and ECS, you can create data connections in Management Center to distinguish data lake services in the development environment from those in the production environment. The data lake is automatically switched during development and production. Therefore, you need to prepare two data lake services (clusters) that have the same version, specifications, components, region, VPC, subnet, and other related configurations.
For example, if your data lake service is an MRS cluster, you need to prepare two MRS clusters with the same version, specifications, components, region, VPC, and subnet. Any change to the configuration of one of the MRS clusters must be synchronized to the other cluster.
- For serverless services (such as DLI), you can configure the mapping between data lake services in the production environment and those in the development environment through environment isolation in Management Center. The data lake is automatically switched during development and production. Therefore, you need to prepare two sets of queue and database resources in the serverless data lake service. You are advised to distinguish them by name suffix.
- If GaussDB(DWS), MRS Hive, and MRS Spark data sources use the same cluster, you must configure database mapping on the Configure Data Source Resource Mapping page to isolate the development and production environments.
Preparing and synchronizing data:
- After creating data lake services, you must create databases, database schemas (required only for DWS), and data tables in the data lake services of the development and production environments based on the project plan (for example, the databases and tables required for data development).
- For clustered data sources (such as MRS, DWS, RDS, MySQL, Oracle, DIS, and ECS), use two clusters, one for the development environment and the other for the production environment. The names of the databases, database schemas (required only for DWS), and data tables in the two environments must be the same.
- For serverless services (such as DLI), you are advised to associate and distinguish the two queues and databases by name suffix (add suffix _dev to the names of the queues and databases in the development environment and add no suffix to those in the production environment). The names of data tables in the development environment must be the same as those in the production environment.
- For DWS, MRS Hive, and MRS Spark data sources that use the same cluster, use two databases to isolate the development and production environments (add suffix _dev to the database for the development environment and add no suffix to the database for the production environment). The names of database schemas (required only for DWS) and data tables in the development environment must be the same as those in the production environment.
- After creating databases, database schemas (required only for DWS), and data tables, you must synchronize data of existing tables (if any) between the two data lake services.
- Existing data in data lakes: Use data migration services such as CDM and DRS to synchronize data in batches between data lakes.
- Data to be migrated from the data source: Use peering jobs of data migration services such as CDM and DRS to synchronize data between the data lake service of the production environment and that of the development environment.
|
|
Creating data connections in enterprise mode |
You must create data connections for all data lake engines.
For clustered data sources that use different clusters, you can create a data connection between DataArts Studio and the data lake of the development environment and a data connection between DataArts Studio and the data lake of the production environment at the same time.
For details, see Creating a DataArts Studio Data Connection. |
|
Configuring environment isolation for a workspace in enterprise mode |
Configure DLI queue and DB mapping to isolate the development and production environments.
For the DWS, MRS Hive, and MRS Spark data sources, if you select the same cluster when creating a data connection, you need to configure two databases for the same data lake service to isolate the development environment from the production environment. For details, see DB Configuration.
For the DLI data source, you can configure two DLI queues and databases to isolate the production environment from the development environment. For details, see Configuring Environment Isolation for a DataArts Studio Workspace in Enterprise Mode. |
