Updated on 2023-07-19 GMT+08:00

Log Transfer

This section describes the restrictions on log transfer.

Table 1

Category

Item

Description

Remarks

Log transfer to OBS

Transfer tasks for a log stream

A log stream can have only one task for transferring logs to OBS.

N/A

Log transfer interval

2 minutes, 5 minutes, 30 minutes, 1 hour, 3 hours, 6 hours, 12 hours

N/A

Data size of each log transfer task

0 MB to 2 GB

N/A

Transfer rate threshold

100 MB/s The transfer may fail if this limit is exceeded.

N/A

Log transfer delay

10 minutes

For example, if the transfer interval is 30 minutes and the transfer starts at 8:30, transfer files will be generated at 8:40 at the latest.

N/A

Target bucket

Standard buckets are supported. Parallel file systems are not supported.

N/A

Log transfer to DIS

Transfer tasks for a log stream

A log stream can have only one task for transferring logs to DIS.

N/A

Log transfer interval

Real time

N/A

Data size of each log transfer task

N/A

N/A

Log transfer delay

N/A

N/A

Transfer rate threshold

Same as the maximum write rate of the relevant DIS stream.

The transferred data will be unstable if this limit is exceeded.

N/A

Log transfer to DMS

Transfer tasks for a log stream

A log stream can have only one task for transferring logs to DMS.

N/A

Log transfer interval

Real time

N/A

Data size of each log transfer task

N/A

N/A

Log transfer delay

N/A

N/A

Transfer rate threshold

Same as the upper traffic limit of the relevant DMS (Kafka) cluster.

The transferred data will be unstable if this limit is exceeded.

N/A

Log transfer to GaussDB(DWS)

Transfer tasks for a log stream

A log stream can have only one task for transferring logs to GaussDB(DWS).

N/A

Log transfer interval

1 minute

N/A

Data size of each log transfer task

< 5 MB

N/A

Log transfer delay

5 minutes

For example, if the transfer starts at 8:30, transfer files will be generated at 8:35 at the latest.

N/A

Transfer rate threshold

40 MB/s

The transferred data will be unstable if this limit is exceeded.

N/A

Data reliability

If the format of a batch of data is valid, the data is transferred at least once. However, if the GaussDB(DWS) cluster is heavily loaded or a network error occurs, the write response will time out, which may cause duplicate data. In this case, data accuracy (Exactly Once) cannot be ensured.

N/A

Table structure change

  • Adding non-mandatory columns to DWS tables does not affect log transfer.
  • Adding mandatory columns to DWS tables during log delivery will cause a data write failure.
  • Deleting columns that contain transfer rules from DWS tables during log delivery will cause a data write failure.

N/A

Invalid data columns

The common scenarios include mismatch and type conversion failure. This batch of data will not be written to GaussDB(DWS), while other batches will be written normally.

N/A

Oversized data columns

The common scenarios include long string and varchar type data. This batch of data will not be written to GaussDB(DWS), while other batches will be written normally.

N/A