Using Logstash for Data Migration

Procedure

The following describes how to use a Logstash cluster to migrate data from one Elasticsearch cluster to another.

Before starting to migrate data, make the necessary preparations. For details, see Preparations.

1. Step 1: Obtaining Elasticsearch Cluster Information: Obtain the addresses of the source and destination Elasticsearch clusters.
2. Step 2: Creating a Logstash Cluster: Create a Logstash cluster for migrating data between Elasticsearch clusters.
3. Step 3: Configuring a Data Migration Task: Configure an Elasticsearch cluster migration task for the Logstash cluster.
4. Step 4: Starting the Migration Task: Start the migration task in the Logstash cluster.
5. Step 5: Stopping Tasks: After data migration is complete, stop the migration task.
6. Step 6: Deleting the Cluster: Delete clusters that you no longer need to reclaim resources.

Preparations

• You have registered with Huawei Cloud and performed real-name authentication. Make sure your account is not frozen or in arrears.
  If you do not have a Huawei Cloud account, perform the following operations to create one:

  1. Visit the Huawei Cloud official website.
  2. In the upper-right corner of the page, click Register and complete the registration as prompted.
  3. Select the service agreement and click Enable.
  4. Perform real-name authentication.
     • If your account is an individual account, see Individual Real-Name Authentication.
     • If your account is an enterprise account, see Enterprise Real-Name Authentication.
• The source Elasticsearch cluster (Source-ES) and destination Elasticsearch cluster (Dest-ES) are ready. Both clusters are single-node non-security mode clusters.

Step 1: Obtaining Elasticsearch Cluster Information

Obtain the addresses of the source and destination Elasticsearch clusters. For security-mode clusters, contact the administrator to obtain their usernames and passwords.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, obtain the IP address of the destination cluster from the Private Network Address column. Generally, the IP address format is <host>:<port> or <host>:<port>,<host>:<port>.

   In this example, the address of the source Elasticsearch cluster (Source-ES) is 10.62.179.32:9200, and that of the destination Elasticsearch cluster (Dest-ES) is 10.62.179.33:9200.

   Figure 1 Obtaining IP addresses
   

Step 2: Creating a Logstash Cluster

Create a Logstash cluster for migrating data between Elasticsearch clusters.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Logstash.
3. In the upper right corner, click Create Cluster. The new-version UI for creating a cluster is displayed by default.
   Figure 2 Create Cluster (new version)
   
4. Select the cluster type and version.

   Table 1 Cluster configuration

   Parameter	Example	Description
   Cluster Type	Logstash	Select Logstash.
   Cluster Version	7.10.0	Select a cluster version from the drop-down list.

5. Configure basic settings, including the region, billing mode, and AZs.

   Table 2 Basic settings

   Parameter	Example	Description
   Region	Hong Kong, China	Select the region where the cluster is located. A region is the location of a physical data center. Regions are defined based on their geographical location and network latency. For lower network latency and quicker resource access, select the nearest region.
   AZ	AZ 1	Select AZs associated with the cluster region. An AZ is a physical region where resources use independent power supplies and networks. AZs are physically isolated but interconnected through an internal network. A maximum of two AZs can be configured.
   Billing Mode	Pay-per-use	Billing mode for the cluster, which can be Yearly/Monthly or Pay-per-use. Yearly/Monthly: You prepay for a yearly or monthly subscription. Pay-per-use (postpaid): You will be billed hourly by actual duration of use. Any partial hour of usage will be rounded up to one hour.

6. Configure data nodes.
   The data nodes in a Logstash cluster are used for processing and transmitting data.

   Figure 3 Configuring data nodes
   

   Table 3 Configuring data nodes

   Parameter	Example	Description
   CPU Architecture	x86	Select the CPU architecture of the data nodes.
   Node Specifications	ess.spec-4u8g	Select the specifications of the data nodes. Click Available. On the displayed page, select a flavor that suits your needs. In the node flavor list, vCPUs | Memory indicate the number of vCPUs and memory capacity available for each flavor, and Recommended Storage indicates the supported storage capacity range. The node flavors available may vary depending on the region you select.
   Node Storage Type and Capacity	High I/O 40GB	Select the storage type and capacity of the data nodes. Available EVS disk types vary depending on your region. The node storage capacity is fixed at 40 GB.
   Nodes	1	Set the number of nodes in the cluster.

7. Configure network settings for the cluster, including the VPC, IP address, and security group.
   Figure 4 Configuring network settings
   

   Table 4 Configuring network settings

   Parameter	Example	Description
   VPC	vpc-default	Select a VPC for the cluster for proper network isolation.
   Subnet	subnet-default	Select a subnet for the cluster. A subnet improves network security by providing exclusive network resources that are isolated from other networks. Select a subnet in the current VPC.
   IPv4 Address	Assign automatically	Assign IPv4 addresses to cluster nodes.
   Security Group	default	Select a security group for the cluster. A security group serves as a virtual firewall that provides access control policies for clusters. The selected security group must allow all ports or port 9200 in the inbound direction. Otherwise, the cluster may be inaccessible to external services.

8. Configure cluster management settings, such as the cluster name and enterprise project.

   Table 5 Cluster management

   Parameter	Example	Description
   Cluster Name	Sample-Logstash	User-defined cluster name.
   Add Description	Skip this setting.	Add a description for the cluster for easy recognition.
   Enterprise Project	default	Associate the cluster with an enterprise project. An enterprise project groups cloud resources, so you can manage resources and members by project. The default project is default. If enterprise projects are enabled, you can select an enterprise project from the drop-down list.
   Tags	None	Adding tags to clusters helps you identify and manage your cluster resources. Each cluster can have a maximum of 20 tags.

9. Click Create Now.
10. Return to the cluster list and check the newly created cluster. If the cluster is created successfully, Cluster Status changes to Available.
    Figure 5 Checking the cluster status
    

Step 3: Configuring a Data Migration Task

Configure an Elasticsearch cluster migration task for the Logstash cluster.

1. In the Logstash cluster list, click Sample-Logstash to show the cluster information page.
2. Click the Configuration Center tab.
3. On the Configuration Center page, click Create in the upper-right corner. On the Create Configuration File page that is displayed, edit the configuration file.
   Figure 6 Create Configuration File
   

   Table 6 Parameters for creating a configuration file

   Parameter	Description	Example Value
   Name	User-defined configuration file name. It can contain only letters, digits, hyphens (-), and underscores (_), and must start with a letter. The minimum length is 4 characters.	es-es
   Configuration File Content	Expand System Templates, find elasticsearch, and click Apply in the Operation column. In the Configuration File Content area, configure the configuration file based on comments in the template.	See Table 7 for key configuration items. Use the default settings for others.
   Hidden Content	For items that you enter in this box, the corresponding strings will be replaced with *** in the configurations. Enter sensitive strings that you want to hide, and press Enter. You can enter a maximum of 20 strings, each with a maximum length of 512 bytes.	N/A

   Table 7 Configuration item description

   Configuration Item	Description	Example Value
   hosts	Enter the addresses of the source and destination Elasticsearch clusters in input and output, respectively. For details about how to obtain the cluster addresses, see Step 1: Obtaining Elasticsearch Cluster Information.	input hosts: http://10.62.179.32:9200 output hosts: http://10.62.179.33:9200
   user	Username for accessing the Elasticsearch cluster. This parameter is required for security-mode clusters. For non-security mode clusters, use # to comment out this parameter.	Use # to comment it out.
   password	Password for accessing the Elasticsearch cluster. This parameter is required for security-mode clusters. For non-security mode clusters, use # to comment out this parameter.	Use # to comment it out.
   index	Specifies indexes that need to be migrated. You can use a wildcard.	index*

4. Click Next to configure Logstash pipeline parameters.
   Figure 7 Configuring pipeline parameters
   

   Table 8 Pipeline parameters

   Parameter	Description	Example Value
   pipeline.workers	Number of worker threads that will execute the Filters and Outputs stages of the pipeline in parallel.	4
   pipeline.batch.size	Maximum number of events that a worker thread collects from inputs before attempting to execute its filters and outputs. A larger value is more effective but increases memory overhead.	125
   pipeline.batch.delay	Maximum amount of time (in milliseconds) that a pipeline worker waits for each new event while its current batch is not yet full.	50
   query.type	An internal queue model for event buffering. memory indicates a traditional memory-based queue. persisted indicates a disk-based ACKed persistent queue.	memory

5. Click Create. The system automatically verifies the configuration file. When the configuration file status changes to Available, the creation is successful.
   Figure 8 Configuration file verification
   

Step 4: Starting the Migration Task

Start the configured migration task in the Logstash cluster.

1. On the Configuration Center page, select the newly created configuration file, whose status is Available, and click Start. In the pipeline list, the Events column shows number of tasks processed by each stage of the pipeline.
   Figure 9 Starting a task
   
2. After the data migration is complete, check the data consistency between the source and destination Elasticsearch clusters. For example, run the GET _cat/indices command in the source and destination clusters, separately, to check whether their indexes are consistent.
   1. In the Elasticsearch cluster list, select the source Elasticsearch cluster Source-ES or the destination Elasticsearch cluster Dest-ES, and click Access Kibana in the Operation column to access the Kibana console.
   2. In the Kibana navigation pane on the left, choose Dev Tools.
   3. On the Console page, run the following command to view index information:

      GET _cat/indices

Step 5: Stopping Tasks

After data migration is complete, stop the migration task.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Logstash.
3. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
4. Click the Configuration Center tab.
5. In the cluster list, select the Sample-Logstash cluster, and click Configuration Center in the Operation column.
6. On the Configuration Center page, select the name of the pipeline that has been started, and click Stop All to stop all running tasks. Wait until all pipeline tasks are stopped.
   Figure 10 Stopping tasks
   

Step 6: Deleting the Cluster

After data migration is completed, you may delete clusters created earlier to reclaim resources. Before you start, make sure that all pipeline tasks are stopped.

To ensure data security, before deleting a cluster, stop all ongoing tasks, and back up the necessary files.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Logstash.
3. In the cluster list, locate the Sample-Logstash cluster, and choose More > Delete in the Operation column.
4. In the confirmation dialog box, type in DELETE, and click OK.

Related Documents

Learn how to use Logstash to synchronize data to CSS's Elasticsearch:

• Synchronizing Data from RDS for MySQL to Elasticsearch Through Logstash