Updated on 2024-03-15 GMT+08:00

Creating a Smart Connect Task (Redis)

Scenario

Create a Smart Connect task to replicate source Redis data to the target Redis through a Kafka instance topic.

  • Redis data copy is unavailable for instances created before July 1, 2022.
  • This function is unavailable for single-node instances.
  • Data in the source Redis instance is synchronized to the target Redis instance in real time.

Restrictions

  • A maximum of 18 Smart Connect tasks can be created for an instance.
  • When you copy Redis data, Kafka and Redis must be connected through the intranet. If they are in different VPCs, connect the network.
  • Redis data copy is available only for Redis instances later than Redis 3.0.
  • After a Smart Connect task is created, task parameters cannot be modified.
  • When full synchronization is triggered, data may not be consistent between the source and target. Redundant data may exist, that is, there is more data on the target end than on the source end. This is because data deletion commands such as DELETE run during disconnection cannot be synchronized to the target end. If full synchronization is triggered by reconnection, some data on the target end is not deleted.
  • Smart Connect tasks for replicating Redis data can only be used for DR scenarios. Data is synchronized unidirectionally. Bidirectional synchronization is not supported.
  • Cloud service providers may modify or customize the Redis kernel and architecture (proxy), causing data and connection incompatibility. In some cases, data inconsistency or even losses may occur. Therefore, both the source and target must be DCS instances or use open-source versions that have been evaluated by DCS for compatibility. In addition, the source and target must have the same version, instance type, capacity, and configuration (for example, whether multi-DB is enabled).
  • If the Redis instance is overloaded and the reserved memory is small, Redis full synchronization will be repeatedly triggered, destabilizing services. Therefore, the Redis instance must meet the following conditions: The CPU usage of the source Redis instance must be less than 50%, the memory usage must be less than 50%, repl-backlog must be at least 1 GB, and client-output-buffer must be at least 10% of maxmemory.
  • The number of databases in the source Redis instance cannot exceed what is supported by the target Redis instance.
  • You are advised to perform the migration during off-peak hours, because the migration task connects to the source instance as a replica, triggering master/replica synchronization.
  • Results of the FLUSHDB and FLUSHALL commands will not be synchronized. To clear data, run FLUSHDB or FLUSHALL on all target sub-instances.
  • In a Smart Connect task that replicates Redis data, the target end must be a DCS instance.
  • If the source is a Redis Cluster DCS instance that uses cross-slot Lua scripts with built-in keys, migration is not supported.
  • If the target Redis instance's memory is insufficient and the volatile-lru eviction policy (maxmemory-policy) triggers data eviction, data inconsistency may occur between the source and target ends. However, this issue does not affect the Smart Connect task. To prevent this situation, set the eviction policy of the target Redis instance to noeviction. If the memory on the target Redis instance is insufficient, data cannot be written to it. The Smart Connect task will retry writing data until it succeeds, ensuring that no data is lost due to eviction.
  • If the source Redis has a String larger than 20 MB, Redis data replication may fail. In this case, split big keys before creating a Smart Connect task. To check the key type and size, see Analyzing Big Keys and Hot Keys.
  • If master/standby switchover is performed on the target Redis while the Smart Connect task is running, data may be lost.

Prerequisites

  • You have enabled Smart Connect.
  • A Kafka instance has been created and is in the Running state.
  • A topic has been created.
  • The SYNC, PSYNC, and REPLCONF commands have been enabled on the source Redis.
  • If the source and target Redis instances are not in the same region, the Smart Connect broker must be deployed in the same region as the source Redis instance. That is, the Kafka instance must be in the same region as the source Redis instance.
  • If the source and target Redis instances are not in the same region, dual Direct Connect connections are required to ensure bandwidth and latency. The Direct Connect bandwidth must be greater than Max(Bandwidth of a Redis full synchronization task x Number of tasks, Bandwidth of an incremental synchronization task x Number of tasks) x 150% (50% is reserved). The latency must be under 10 ms.

Procedure

  1. Log in to the management console.
  2. Click in the upper left corner to select a region.

    Select the region where your Kafka instance is located.

  3. Click and choose Middleware > Distributed Message Service for Kafka to open the console of DMS for Kafka.
  4. Click the desired Kafka instance to view its details.
  5. In the navigation pane, choose Smart Connect.
  6. On the displayed page, click Create Task.
  7. For Task Name, enter a unique Smart Connect task name.
  8. For Task Type, select Copy Redis data.
  9. In the Source area, set parameters based on the following table.

    Table 1 Source parameters

    Parameter

    Description

    Configure

    Options:

    • By instance: Select a Redis instance. (Recommended)
    • Manually: Manually specify Redis information. Proxy Cluster instances cannot be manually specified.

      If the source Redis is a DCS instance, contact technical support to enable the PSYNC command.

    Redis Instance

    Set this parameter when Configure is set to By instance.

    Select an existing Redis instance from the drop-down list.

    Architecture

    Set this parameter when Configure is set to Manually.

    Options:

    • Single-node: Select this one if the source Redis is the master/standby or single-node type.
    • Cluster: Select this one if the source Redis is the Redis Cluster type.

    Connection Address/Port

    Set this parameter when Configure is set to Manually.

    Enter the IP address and port number for connecting to the Redis instance.

    Password

    Enter the password, if any, of the Redis instance.

    • Redis data copy is available only for Redis instances later than Redis 3.0.
    • The source Redis cannot be a read/write splitting instance.
    • Kafka and Redis can only be connected through the intranet. If they are in different VPCs, connect the network.

  10. Click Check to test connectivity of the source Redis.

    • If Configure is set to By instance, perform the following steps:
      1. In the Test results dialog box that is displayed, enable psync (and vpcep, if applicable), and click Confirm.

        Enable psync in the Test results dialog box for a non-Proxy Cluster instance.

        Figure 1 Non-Proxy Cluster Redis instance

        Enable psync and vpcep in the Test results dialog box for a Proxy Cluster instance.

        Figure 2 Proxy Cluster Redis instance
      2. Click Check again. If "Connectivity test passed." is displayed, the source Redis can be connected to Kafka.
    • If Configure is set to Manually, the source Redis can be connected to Kafka when "Connectivity test passed." is displayed.
    Figure 3 Connectivity check passed

  11. In the Topics area, select a topic. The source Redis data will be copied to this topic.
  12. In the Target area, set parameters based on the following table.

    Table 2 Target parameters

    Parameter

    Description

    Configure

    Options:

    • By instance: Select a Redis instance. (Recommended)
    • Manually: Manually specify Redis information. Proxy Cluster instances cannot be manually specified.

    Redis Instance

    Set this parameter when Configure is set to By instance.

    Select an existing Redis instance from the drop-down list.

    Architecture

    Set this parameter when Configure is set to Manually.

    Options:

    • Single-node: Select this one if the source Redis is the master/standby, single-node, or read/write splitting type.
    • Cluster: Select this one if the source Redis is the Redis Cluster type.

    Connection Address/Port

    Set this parameter when Configure is set to Manually.

    Enter the IP address and port number for connecting to the Redis instance.

    Password

    Enter the password, if any, of the Redis instance.

    • Redis data copy is available only for Redis instances later than Redis 3.0.
    • Kafka and Redis can only be connected through the intranet. If they are in different VPCs, connect the network.

  13. Click Check. If "Connectivity test passed." is displayed, the source Redis can be connected to Kafka.
  14. Click Create. The Smart Connect task list page is displayed. The message "Task xxx was created successfully." is displayed in the upper right corner of the page.