Log Transfer

Updated on 2023-07-19 GMT+08:00

View PDF

This section describes the restrictions on log transfer.

**Table 1**
Category	Item	Description	Remarks
Log transfer to OBS	Transfer tasks for a log stream	A log stream can have only one task for transferring logs to OBS.	N/A
	Log transfer interval	2 minutes, 5 minutes, 30 minutes, 1 hour, 3 hours, 6 hours, 12 hours	N/A
	Data size of each log transfer task	0 MB to 2 GB	N/A
	Transfer rate threshold	100 MB/s The transfer may fail if this limit is exceeded.	N/A
	Log transfer delay	10 minutes For example, if the transfer interval is 30 minutes and the transfer starts at 8:30, transfer files will be generated at 8:40 at the latest.	N/A
	Target bucket	Standard buckets are supported. Parallel file systems are not supported.	N/A
Log transfer to DIS	Transfer tasks for a log stream	A log stream can have only one task for transferring logs to DIS.	N/A
	Log transfer interval	Real time	N/A
	Data size of each log transfer task	N/A	N/A
	Log transfer delay	N/A	N/A
	Transfer rate threshold	Same as the maximum write rate of the relevant DIS stream. The transferred data will be unstable if this limit is exceeded.	N/A
Log transfer to DMS	Transfer tasks for a log stream	A log stream can have only one task for transferring logs to DMS.	N/A
	Log transfer interval	Real time	N/A
	Data size of each log transfer task	N/A	N/A
	Log transfer delay	N/A	N/A
	Transfer rate threshold	Same as the upper traffic limit of the relevant DMS (Kafka) cluster. The transferred data will be unstable if this limit is exceeded.	N/A
Log transfer to GaussDB(DWS)	Transfer tasks for a log stream	A log stream can have only one task for transferring logs to GaussDB(DWS).	N/A
	Log transfer interval	1 minute	N/A
	Data size of each log transfer task	< 5 MB	N/A
	Log transfer delay	5 minutes For example, if the transfer starts at 8:30, transfer files will be generated at 8:35 at the latest.	N/A
	Transfer rate threshold	40 MB/s The transferred data will be unstable if this limit is exceeded.	N/A
	Data reliability	If the format of a batch of data is valid, the data is transferred at least once. However, if the GaussDB(DWS) cluster is heavily loaded or a network error occurs, the write response will time out, which may cause duplicate data. In this case, data accuracy (Exactly Once) cannot be ensured.	N/A
	Table structure change	Adding non-mandatory columns to DWS tables does not affect log transfer. Adding mandatory columns to DWS tables during log delivery will cause a data write failure. Deleting columns that contain transfer rules from DWS tables during log delivery will cause a data write failure.	N/A
	Invalid data columns	The common scenarios include mismatch and type conversion failure. This batch of data will not be written to GaussDB(DWS), while other batches will be written normally.	N/A
	Oversized data columns	The common scenarios include long string and varchar type data. This batch of data will not be written to GaussDB(DWS), while other batches will be written normally.	N/A