Updated on 2024-10-23 GMT+08:00

To Elasticsearch

Table 1 Parameter description

Type

Parameter

Description

Example Value

Basic parameters

Index

Elasticsearch index, which is similar to the name of a relational database.CDM supports automatic creation of indexes and field types. The index and field type names can contain only lowercase letters.

index

Type

Elasticsearch type, which is similar to the table name of a relational database. The type name can contain only lowercase letters.

NOTE:

Elasticsearch 7.x and later versions do not support custom types. Instead, only the _doc type can be used. In this case, this parameter does not take effect even if it is set.

type

Operation

Operation type

  • INDEX: No primary key is required. Elasticsearch generates IDs so that data is written to a new file with a unique ID for each write operation.
  • CREATE: A primary key needs to be specified. If the primary key already exists, the write operation fails.
  • UPDATE: A primary key needs to be specified. If the primary key already exists, the original data is overwritten.
  • UPSERT: A primary key needs to be specified. If the primary key already exists, the operation is the same as that of UPDATE. If no primary key exists, data is written to a new document.

UPSERT

Primary Key Mode

This parameter is available when Operator is UPSERT, UPDATE, or CREATE.

  • Single primary key: Select a primary key and write its value to the ID.
  • Composite primary key: Select multiple primary keys and write their values to the ID using primary key delimiters.
  • No primary key: This value is available only when Operator is set to CREATE. You do not need to specify a primary key. The destination automatically generates an ID as the primary key.

Single primary key

Clear Data Before Import

Whether to delete data when the current task already exists in the index.

  • Yes: Data in the index needs to be deleted.
  • No: Existing data is retained before new data is written.

No

Primary Key Delimiter

This parameter is available when Primary Key Mode is Composite primary key. It separates the primary keys to be written to the ID.

_

Advanced attributes

Pipeline ID

This parameter is available only after a pipeline ID is created in Kibana. It is used to convert the data format using the data transformation pipeline of Elasticsearch after data is transferred to Elasticsearch.

pipeline_id

Write ES with Routing

If you enable this function, a column can be written to Elasticsearch as a route.

NOTE:

Before enabling this function, create indexes at the destination to improve the query efficiency.

No

Routing Column

This parameter is available when Write ES with Routing is set to Yes. It specifies the destination routing column. If the destination index exists but the column information cannot be obtained, you can manually enter the column. The routing column can be empty. If it is empty, no routing value is specified for the data written to Elasticsearch.

value1

Periodically Create Index

For streaming jobs that continuously write data to Elasticsearch, CDM periodically creates indexes and writes data to the indexes, which helps you delete expired data. The indexes can be created based on the following periods:
  • Every hour: CDM creates indexes on the hour. The new indexes are named in the format of Index name+Year+Month+Day+Hour, for example, index2018121709.
  • Every day: CDM creates indexes at 00:00 every day. The new indexes are named in the format of Index name+Year+Month+Day, for example, index20181217.
  • Every week: CDM creates indexes at 00:00 every Monday. The new indexes are named in the format of Index name+Year+Week, for example, index201842.
  • Every month: CDM creates indexes at 00:00 on the first day of each month. The new indexes are named in the format of Index name+Year+Month, for example, index201812.
  • Do not create: Do not create indexes periodically.

When extracting data from a file, you must configure a single extractor, which means setting Concurrent Extractors to 1. Otherwise, this parameter is invalid.

Every hour

Commits

Size of data to be submitted at a time

10000

Retries

Number of retries upon a request failure. A maximum of 10 retries are allowed.

3