Different Ways to Import Data to an Elasticsearch Cluster

Introduction

Elasticsearch clusters support multiple data import methods, as listed in Table 1. Select one that fits your needs the best. Before starting to import data, determine whether to enhance the data import performance of Elasticsearch clusters first. For details, see Enhancing the Data Import Performance of Elasticsearch Clusters.

**Table 1** Different ways to import data to an Elasticsearch cluster
Data Import Method	Scenario	Supported Data Formats	Details
Logstash data processing pipeline	Open-source Logstash offers a server-side, real-time data processing pipeline, which supports data ingestion from multiple sources. It can be used to collect various data, such as logs, monitoring data, and metrics.	JSON, CSV, and text	Using In-house Built Logstash to Import Data to Elasticsearch
Open-source Elasticsearch API	Open-source Elasticsearch APIs can be used to import data. This method is flexible, as you can write your own application code.	JSON	Using Open Source Elasticsearch APIs to Import Data to Elasticsearch
Cloud Data Migration (CDM)	You can use CDM for batch data migration. For example, if data is stored in OBS or an Oracle database, CDM is recommended.	JSON	Using CDM to Import Data to Elasticsearch
Data Replication Service (DRS)	DRS can be used for online database migration and real-time data synchronization.	Relational Database Service (RDS)	Using DRS to Import Data from a Database to Elasticsearch

Using DRS to Import Data from a Database to Elasticsearch

DRS is an easy-to-use, stable, and efficient cloud service for online database migration and real-time database synchronization. Real-time data synchronization refers to the real-time replication of data from one source to another while ensuring data consistency.

DRS can be used to import data from multiple types of relational databases to Elasticsearch clusters. For details about the supported software versions for source databases and destination clusters, see Table 2.

**Table 2** Supported versions for using DRS to import data from a database to Elasticsearch
Scenario	Source DB	Destination Elasticsearch Cluster	Details
Importing data from an RDS for MySQL database to a CSS Elasticsearch cluster	RDS for MySQL 5.5, 5.6, 5.7, or 8.0	Elasticsearch 5.5, 6.2, 6.5, 7.1, 7.6, 7.9, or 7.10	(Real-Time Synchronization) From MySQL to CSS/ES
Importing data from a GaussDB(for MySQL) database to a CSS Elasticsearch cluster	GaussDB(for MySQL) active/standby instances	Elasticsearch 5.5, 6.2, 6.5, 7.1, 7.6, 7.9, or 7.10	(Real-Time Synchronization) From GaussDB(for MySQL) to CSS/ES
Importing data from an in-house built MySQL database to a CSS Elasticsearch cluster	MySQL database 5.5, 5.6, 5.7, or 8.0 created on a local server or ECS	Elasticsearch 5.5, 6.2, 6.5, 7.1, 7.6, 7.9, or 7.10	(Real-Time Synchronization) From MySQL to CSS/ES