Using CDM to Import Data from OBS to Elasticsearch

You can use the CDM-provided wizard to import data stored in OBS to Elasticsearch in CSS. Data files can be in the JSON or CSV format.

Figure 1 shows the data transmission process.

Figure 1 Process of using CDM to import OBS data to Elasticsearch

Procedure

Log in to the OBS management console.
Create an OBS bucket for storing data.
For details, see "Creating a Bucket" in the Object Storage Service Console Operation Guide.

The OBS bucket must be in the same region as the cluster.

Upload the data file to the OBS bucket.

For details, see "Uploading a File" in the Object Storage Service Console Operation Guide.

For example, save the following data as a JSON file and upload the file to the OBS bucket.

{"productName":"Latest art shirts for women in autumn 2017","size":"L"}
{"productName":"Latest art shirts for women in autumn 2017","size":"M"}
{"productName":"Latest art shirts for women in autumn 2017","size":"S"}
{"productName":"Latest jeans for women in spring 2018","size":"M"}
{"productName":"Latest jeans for women in spring 2018","size":"S"}
{"productName":"Latest casual pants for women in spring 2017","size":"L"}
{"productName":"Latest casual pants for women in spring 2017","size":"S"}

Log in to the CSS management console.
In the navigation pane on the left, choose Clusters > Elasticsearch to switch to the Clusters page.
From the cluster list, locate the row that contains the cluster to which you want to import data, and click Access Kibana in the Operation column.
In the Kibana navigation pane on the left, choose Dev Tools.

On the Console page, run the related command to create an index for the data to be stored and specify a custom mapping to define the data type:

If there is an available index in the cluster where you want to import data, this step is not required. If there is no available index, create an index by referring to the following sample code.

For example, on the Console page, run the following command to create index demo and specify a user-defined mapping to define the data type:

Versions earlier than 7.x

PUT /demo
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "products": {
      "properties": {
        "productName": {
          "type": "text",
          "analyzer": "ik_smart"
        },
        "size": {
          "type": "keyword"
        }
      }
    }
  }
}

Versions later than 7.x

PUT /demo
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
      "properties": {
        "productName": {
          "type": "text",
          "analyzer": "ik_smart"
        },
        "size": {
          "type": "keyword"
        }
      }
    }
  }

The command is successfully executed if the following information is displayed.

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "demo"
}

Log in to the CDM management console.
Purchase a CDM cluster.
For details, see "Creating a Cluster" in the Cloud Data Migration User Guide.
Create a link between CDM and CSS.
For details, see "Creating a Link" in the Cloud Data Migration User Guide.
Create a link between CDM and OBS.
For details, see "Creating a Link" in the Cloud Data Migration User Guide.
Create a job on the CDM cluster and migrate the data in the OBS bucket to the target cluster in CSS.
For details, see "Table/File Migration" in the Cloud Data Migration User Guide.

On the Console page of Kibana, search for the imported data.

On the Console page of Kibana, run the following command to search for data. View the search results. If the searched data is consistent with the imported data, the data has been imported successfully.

GET demo/_search

The command is successfully executed if the following information is displayed.

{
  "took": 18,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 7,
    "max_score": 1,
    "hits": [
      {
        "_index": "demo",
        "_type": "products",
        "_id": "g6UepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"L"}""",
          "productName": """{"productName":"Latest art shirts for women in autumn 2017""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "hKUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"M"}""",
          "productName": """{"productName":"Latest art shirts for women in autumn 2017""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "haUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"S"}""",
          "productName": """{"productName":"Latest art shirts for women in autumn 2017""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "hqUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"M"}""",
          "productName": """{"productName":"Latest jeans for women in autumn 2018""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "h6UepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"S"}""",
          "productName": """{"productName":"Latest jeans for women in autumn 2018""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "iKUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"L"}""",
          "productName": """{"productName":"Latest casual pants for women in autumn 2017""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "iaUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"S"}""",
          "productName": """{"productName":"Latest casual pants for women in autumn 2017""""
        }
      }
    ]
  }
}