Different Ways to Ingest Data into an OpenSearch Cluster

Introduction

OpenSearch clusters support multiple data ingestion methods, as listed in Table 1. Select one that fits your needs the best. Before starting to ingest data, determine whether to enhance the data ingestion performance of OpenSearch clusters first. For details, see Enhancing the Data Ingestion Performance of OpenSearch Clusters.

**Table 1** Different ways to ingest data into an OpenSearch cluster
Data Ingestion Method	Scenario	Supported Data Formats	Details
Open-source Logstash	Open-source Logstash offers a server-side, real-time data processing pipeline, which supports data ingestion from multiple sources. It can be used to ingest various types of data, such as logs, monitoring data, and metrics.	JSON, CSV, and text	Using In-house Built Logstash to Ingest Data into an OpenSearch Cluster
Open-source OpenSearch API	Open-source Elasticsearch APIs can be used to ingest data. This method is flexible, as you can write your own application code.	JSON	Using Open Source OpenSearch APIs to Import Data to an OpenSearch Cluster
Cloud Data Migration (CDM)	You can use CDM for batch data migration. For example, if data is stored in OBS or an Oracle database, CDM is recommended.	JSON	Using CDM to Import Data to an OpenSearch Cluster
Data Replication Service (DRS)	DRS can be used for online database migration and real-time data synchronization.	Relational Database Service (RDS)	Using DRS to Ingest Data from a Database to OpenSearch

Using DRS to Ingest Data from a Database to OpenSearch

DRS is an easy-to-use, stable, and efficient cloud service for online database migration and real-time database synchronization. Real-time data synchronization refers to the real-time replication of data from one source to another while ensuring data consistency.

DRS can be used to ingest data from multiple types of relational databases to OpenSearch clusters. For details about the supported software versions for source databases and destination clusters, see Table 2.

**Table 2** Using DRS to ingest data from a database to an OpenSearch cluster
Scenario	Source DB	Destination OpenSearch Cluster	Details
Ingesting data from an RDS for MySQL database into a CSS OpenSearch cluster	RDS for MySQL 5.5, 5.6, 5.7, or 8.0	OpenSearch 1.3.6	(Real-Time Synchronization) From MySQL to CSS/ES
Ingesting data from a TaurusDB database into a CSS OpenSearch cluster	Primary/standby TaurusDB instances	OpenSearch 1.3.6	(Real-Time Synchronization) From TaurusDB to CSS/Elasticsearch
Ingesting data from an in-house built MySQL database into a CSS OpenSearch cluster	MySQL database 5.5, 5.6, 5.7, or 8.0 created on a local server or ECS	OpenSearch 1.3.6	(Real-Time Synchronization) From MySQL to CSS/ES