Using Logstash for Data Migration

Procedure

The following describes how to use a Logstash cluster to migrate data from one Elasticsearch cluster to another.

Before starting to migrate data, make the necessary preparations. For details, see Preparations.

1. Step 1: Obtaining Elasticsearch Cluster Information: Obtain the addresses of the source and destination Elasticsearch clusters.
2. Step 2: Creating a Logstash Cluster: Create a Logstash cluster for migrating data between Elasticsearch clusters.
3. Step 3: Configuring a Data Migration Task: Configure an Elasticsearch cluster migration task for the Logstash cluster.
4. Step 4: Starting the Migration Task: Start the migration task in the Logstash cluster.
5. Step 5: Stopping the Task: After data migration is complete, stop the migration task.

Preparations

• The source Elasticsearch cluster (Source-ES) and destination Elasticsearch cluster (Dest-ES) are ready. Both clusters are single-node non-security mode clusters.
• Confirm the indexes to be migrated in the source Elasticsearch cluster. In our example, the indexes whose name starts with index are to be migrated.

Step 1: Obtaining Elasticsearch Cluster Information

Obtain the addresses of the source and destination Elasticsearch clusters. For security-mode clusters, contact the administrator to obtain their usernames and passwords.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, obtain the IP address of the destination cluster from the Private Network Address column. Generally, the IP address format is <host>:<port> or <host>:<port>,<host>:<port>.

   In this example, the address of the source Elasticsearch cluster (Source-ES) is 10.62.179.32:9200, and that of the destination Elasticsearch cluster (Dest-ES) is 10.62.179.33:9200.

Step 2: Creating a Logstash Cluster

Create a Logstash cluster for migrating data between Elasticsearch clusters.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Logstash.
3. Click Create Cluster in the upper-right corner. The Create Cluster page is displayed.
4. Configure Billing Mode and AZ for the cluster.

   Table 1 Billing mode and AZ parameters

   Parameter	Description	Example Value
   Billing Mode	Select Yearly/Monthly or Pay-per-use. Yearly/monthly: You pay for the cluster by year or month, in advance. The subscription duration ranges from one month to three years. If you plan to use a cluster for more than nine months, you are advised to purchase a yearly package for a better price. Pay-per-use: You will be billed hourly by actual duration of use. Any partial hour of usage will be rounded up to one hour.	Pay-per-use
   Region	Select the region where the cluster is located. ECSs in different regions cannot communicate with each other over an intranet. For lower network latency and quicker resource access, select the nearest region.	xxx
   AZ	Select AZs associated with the cluster region. A maximum of two AZs can be configured.	AZ 1

5. Configure basic cluster information.
   Figure 1 Configuring cluster information
   

   Table 2 Basic configuration parameters

   Parameter	Description	Example Value
   Cluster Type	Select Logstash.	Logstash
   Cluster Version	Select a cluster version from the drop-down list.	7.10.0
   Cluster Name	User-defined cluster name.	Sample-Logstash
   Cluster Description	Add a description for the cluster for easy recognition.	/

6. Configure the cluster's node specifications.
   Figure 2 Configuring the cluster's node specifications
   

   Table 3 Specification parameters

   Parameter	Description	Example Value
   Nodes	Number of nodes in a cluster, in the range 1 to 100.	1
   CPU Architecture	Only x86 is supported.	x86
   Node Specifications	Select the specifications of cluster nodes.	ess.spec-4u8g
   Node Storage Type	Select the storage type of cluster nodes.	High I/O
   Node Storage Capacity	Set the storage capacity of a single cluster node. The default value is 40 GB.	40GB

7. Set the enterprise project.

   When creating a CSS cluster, you can bind an enterprise project to the cluster if you have enabled the enterprise project function. In this example, default, the default enterprise project, is selected.

8. Click Next: Network to configure the cluster network.
   Figure 3 Configuring networking
   

   Table 4 Network configuration parameters

   Parameter	Description	Example Value
   VPC	Specify a VPC to isolate the cluster's network. Select the VPC used by the Elasticsearch clusters. NOTE: The VPC must contain CIDRs. Otherwise, cluster creation will fail. By default, a VPC will contain CIDRs.	vpc-default
   Subnet	A subnet provides dedicated network resources that are isolated from other networks, improving network security.	subnet-default
   Security Group	A security group serves as a virtual firewall that provides access control policies for clusters. NOTE: To enable cluster access, ensure that port 9200 is allowed by the security group.	default

9. Click Next: Configure Advanced Settings.

   This cluster is used only for getting started. There is no need to enable advanced settings.

10. Click Next: Confirm. Check the configuration and click Next to create a cluster.
11. Click Back to Cluster List to switch to the Clusters page. The cluster you created is now in the cluster list and its status is Creating. If the cluster is successfully created, its status changes to Available.
    Figure 4 Creating a cluster
    

Step 3: Configuring a Data Migration Task

Configure an Elasticsearch cluster migration task for the Logstash cluster.

1. In the Logstash cluster list, click Sample-Logstash to show the cluster information page.
2. Click the Configuration Center tab.
3. On the Configuration Center page, click Create in the upper-right corner. On the Create Configuration File page that is displayed, edit the configuration file.
   Figure 5 Create Configuration File
   

   Table 5 Parameters for creating a configuration file

   Parameter	Example	Description
   Name	es-es	User-defined configuration file name. It can contain only letters, digits, hyphens (-), and underscores (_), and must start with a letter. The minimum length is 4 characters.
   Configuration File Content	See Table 6 for key configuration items. Use the default settings for others.	Expand System Templates, find elasticsearch, and click Apply in the Operation column. In the Configuration File Content area, configure the configuration file based on comments in the template.
   Hidden Content	N/A	For items that you enter in this box, the corresponding strings will be replaced with *** in the configurations. Enter sensitive strings that you want to hide, and press Enter. You can enter a maximum of 20 strings, each with a maximum length of 512 bytes.

   Table 6 Configuration item description

   Configuration Item	Example	Description
   hosts	input hosts: http://10.62.179.32:9200 output hosts: http://10.62.179.33:9200	Enter the addresses of the source and destination Elasticsearch clusters in input and output, respectively. For details about how to obtain the cluster addresses, see Step 1: Obtaining Elasticsearch Cluster Information.
   user	Use # to comment it out.	Username for accessing the Elasticsearch cluster. This parameter is required for security-mode clusters. For non-security mode clusters, use # to comment out this parameter.
   password	Use # to comment it out.	Password for accessing the Elasticsearch cluster. This parameter is required for security-mode clusters. For non-security mode clusters, use # to comment out this parameter.
   index	index*	Specifies indexes that need to be migrated. You can use a wildcard.

4. Click Next to configure Logstash pipeline parameters.
   Figure 6 Configuring pipeline parameters
   

   Table 7 Pipeline parameters

   Parameter	Example	Description
   pipeline.workers	4	Number of worker threads that will execute the Filters and Outputs stages of the pipeline in parallel.
   pipeline.batch.size	125	Maximum number of events that a worker thread collects from inputs before attempting to execute its filters and outputs. A larger value is more effective but increases memory overhead.
   pipeline.batch.delay	50	Maximum amount of time (in milliseconds) that a pipeline worker waits for each new event while its current batch is not yet full.
   query.type	memory	An internal queue model for event buffering. memory indicates a traditional memory-based queue. persisted indicates a disk-based ACKed persistent queue.

5. Click Create. The system automatically verifies the configuration file. When the configuration file status changes to Available, the creation is successful.
   Figure 7 Configuration file verification
   

Step 4: Starting the Migration Task

Start the configured migration task in the Logstash cluster.

1. On the Configuration Center page, select the newly created configuration file, whose status is Available, and click Start. In the pipeline list, the Events column shows number of tasks processed by each stage of the pipeline.
   Figure 8 Starting a task
   
2. After the data migration is complete, check the data consistency between the source and destination Elasticsearch clusters. For example, run the GET _cat/indices command in the source and destination clusters, separately, to check whether their indexes are consistent.
   1. In the Elasticsearch cluster list, select the source Elasticsearch cluster Source-ES or the destination Elasticsearch cluster Dest-ES, and click Access Kibana in the Operation column to access the Kibana console.
   2. In the Kibana navigation pane on the left, choose Dev Tools.

      The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

   3. On the Console page, run the following command to view index information:

      GET _cat/indices

Step 5: Stopping the Task

After data migration is complete, stop the migration task.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Logstash.
3. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
4. Click the Configuration Center tab.
5. In the cluster list, select the Sample-Logstash cluster, and click Configuration Center in the Operation column.
6. On the Configuration Center page, select the name of the pipeline that has been started, and click Stop All to stop all running tasks. Wait until all pipeline tasks are stopped.
   Figure 9 Stopping tasks
   

Follow-up Operations

After data migration is completed, you may delete clusters created earlier to reclaim resources. Before you start, make sure that all pipeline tasks are stopped.

To ensure data security, before deleting a cluster, stop all ongoing tasks, and back up the necessary files.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Logstash.
3. In the cluster list, locate the Sample-Logstash cluster, and choose More > Delete in the Operation column.
4. In the confirmation dialog box, type in DELETE, and click OK.
5. (Optional) Delete the source and destination Elasticsearch clusters following this same procedure if they are no longer needed.