Help Center/ Cloud Search Service/ User Guide/ Using Logstash for Data Migration/ Configuring a Logstash Migration Task/ Starting a Logstash Migration Task
Updated on 2025-01-23 GMT+08:00
View PDF
Share

Starting a Logstash Migration Task

Scenario

To start a migration task with a Logstash cluster, perform the following steps:

  1. Test the network connectivity between the data source and destination and the Logstash cluster. For details, see Testing Connectivity.
  2. Start a Logstash migration task by starting a configuration file.
  3. Monitor the migration task through metrics, operation records, and run logs. For details, see Monitoring Migration Tasks.

Constraints

  • There can be up to 50 Running configuration files in the pipeline list of a Logstash cluster.
  • Configuration files that use the logstash stdin plugin cannot use the Hot Start option.
  • If hot-starting a configuration file fails and the Logstash process exits unexpectedly, the cluster will try to restart the original Logstash process. Exercise caution when using the Hot Start option.

Testing Connectivity

Before using a Logstash cluster to migrate data, test the network connectivity between the data source and destination and the Logstash cluster.

  1. Log in to the CSS management console.
  2. In the navigation pane on the left, choose Clusters > Logstash. The cluster list is displayed.
  3. In the displayed cluster list, locate the row that contains the target cluster, and click Configuration Center in the Operation column. The Configuration Center page is displayed.

    Alternatively, in the cluster list, click the cluster name to go to the cluster information page. In the navigation pane on the left, choose Configuration Center.

  4. On the Configuration Center page, click Test Connectivity.
  5. In the Test Connectivity dialog box, enter the IP address and port number of the data source and destination, and click Test.

    You can test a maximum of 10 IP addresses at a time. You can click Add to add more IP addresses and click Test at the bottom to test connectivity to multiple IP addresses at a time.

    Figure 1 Test Connectivity

    If Available is displayed, the network is connected. If the network is disconnected, configure routes for the Logstash cluster to connect the clusters. For details, see Configuring Routes for a Logstash Cluster.

Starting a Configuration File

If there is no Running task in the pipeline list, start the Logstash migration task by starting a configuration file. You can select multiple configuration files to start them all at once. A maximum of 50 configuration files can be selected at a time.

  1. In the configuration file list, select configuration files you want to start, and click Start in the upper left corner.
  2. In the Start Logstash dialog box, select Keepalive if necessary.

    When Keepalive is enabled, a daemon process will be configured on each node. If the Logstash service becomes faulty, the daemon process will try to rectify the fault and restart the service. You are advised to enable Keepalive for services running long-term. Do not enable it for services running only short-term, or your migration tasks may fail due to a lack of source data.

  3. Click OK to start the configuration file and hence the Logstash migration task.

    You can view the started configuration file in the pipeline list.

Hot Starting a Configuration File

If there are Running tasks in the pipeline list, start the Logstash migration task by hot-starting a configuration file. Only one configuration file can be hot-started at a time. That is, only one pipeline can be added at a time.

  1. In the configuration file list, select a configuration file, and click Hot Start in the upper left corner.
  2. In the Hot Start Logstash Configuration dialog box, the Keepalived setting is kept consistent with other tasks in the pipeline list by default and cannot be changed.
  3. Click OK to hot-start the configuration file and hence the Logstash migration task.

    You can view the hot-started configuration file in the pipeline list.

Monitoring Migration Tasks

After a configuration file is started, a task is generated in the pipeline list, where you can monitor the task.

  • In the pipeline list, click Metric Monitoring in the Operation column to go to the Cloud Eye console and view monitoring information.

    When the Events data of a pipeline changes dynamically, the monitoring data changes accordingly. When a pipeline task is being started or stopped, or the Events data is stable, the monitoring data remains unchanged.

    For details about the monitoring metrics supported, see Logstash Pipeline Monitoring Metrics. For how to configure monitoring alarms, see Using Cloud Eye to Monitor Logstash Clusters.

  • You can also click View Operation Record or View Running Log above the pipeline list to view configuration file operation records or the run logs of a migration task.