What Should I Do If the Log Prompts that the Date Format Fails to Be Parsed?

Symptom

When CDM is used to migrate other data sources to CSS, the job fails to be executed and the error message "Unparseable date" is displayed in the log. See Figure 1.

Figure 1 Log output

Possible Cause

CSS has a special processing mechanism on the time field. If the stored time data does not contain the time zone information, Kibana considers the time as the GMT.

The time displayed in the log may be different from the local time. For example, the time displayed in the log is eight hours earlier than the local time in the GMT+08:00 time zone. Therefore, when CDM migrates data to Cloud Search Service, if the index and type are automatically created by CDM (for example, if date_test and test1 of the migration destination highlighted in Figure 2 do not exist in Cloud Search Service, CDM automatically creates the index and type in Cloud Search Service), CDM, by default, sets the format of the time field to the standard format of yyyy-MM-dd HH:mm:ss.SSS Z, for example, 2018-01-08 08:08:08.666 +0800.

Figure 2 Job configuration

When data is imported from another data source to CSS, if the date format in the source data is not the standard format, for example, 2018/01/05 15:15:46, the CDM job fails to be executed, and the log shows that the date format cannot be parsed. You need to configure a field converter on CDM to convert the format of the date field to the required format of CSS.

Solution

1. Edit the job and go to the Map Field tab page. Click the icon for creating a converter in the row of the source field to create a converter. See Figure 3.
   Figure 3 Creating a converter
   
2. Select Expression conversion as the converter. Currently, expression conversion supports functions of the character string and date types. The syntax is similar to the Java character string and time functions. For details about how to compile the expression, see Expression Conversion.
3. In this example, the source time format is yyyy/MM/dd HH:mm:ss. To convert the source time format to yyyy-MM-dd HH:mm:ss.SSS Z, perform the following operations:
   1. Add the time zone information +0800 to the end of the original date character string. The corresponding expression is value+" +0800".
   2. Use the original date format to parse the string to a date object. You can use the DateUtils.parseDate function for parsing. The syntax is DateUtils.parseDate(String value, String format).
   3. Format the date object into a character string in target format by using the DateUtils.format function. The syntax is DateUtils.format(Date date, String format).

   In this example, the complete expression is DateUtils.format(DateUtils.parseDate(value+" +0800","yyyy/MM/dd HH:mm:ss Z"),"yyyy-MM-dd HH:mm:ss.SSS Z"). See Figure 4.

   Figure 4 Configuring the expression
   
4. Save the converter configuration and save and run the job to solve the problem that Cloud Search Service fails to parse the date format.