Help Center> Cloud Search Service> User Guide> Importing Data to Elasticsearch> Using CDM to Import Data from OBS to Elasticsearch
View PDF

Using CDM to Import Data from OBS to Elasticsearch

You can use the CDM-provided wizard to import data stored in OBS to Elasticsearch in CSS. Data files can be in the JSON or CSV format.

Figure 1 shows the data transmission process.

Figure 1 Process of using CDM to import OBS data to Elasticsearch

Procedure

  1. Log in to the OBS management console.
  2. Create an OBS bucket for storing data.

    For details, see Creating a Bucket in the Object Storage Service Console Operation Guide.

  3. Upload the data file to the OBS bucket.

    For details, see Uploading a File in the Object Storage Service Console Operation Guide.

    For example, save the following data as a JSON file and upload the file to the OBS bucket.

    {"productName":"Latest art shirts for women in autumn 2017","size":"L"}
{"productName":"Latest art shirts for women in autumn 2017","size":"M"}
{"productName":"Latest art shirts for women in autumn 2017","size":"S"}
{"productName":"Latest jeans for women in spring 2018","size":"M"}
{"productName":"Latest jeans for women in spring 2018","size":"S"}
{"productName":"Latest casual pants for women in spring 2017","size":"L"}
{"productName":"Latest casual pants for women in spring 2017","size":"S"}
  4. Log in to the CSS management console.
  5. In the left navigation pane, click Clusters to switch to the Clusters page.
  6. From the cluster list, locate the row where the cluster to which you want to import data resides, and click Kibana in the Operation column.
  7. In the left navigation pane of Kibana, click Dev Tools. Click Get to work to switch to the Console page.
  8. On the Console page, run the related command to create an index for the data to be stored and specify a custom mapping to define the data type:

    If there is an available index in the cluster where you want to import data, this step is not required. If there is no available index, create an index by referring to the following sample code.

    For example, on the Console page, run the following command to create index demo and specify a user-defined mapping to define the data type: 
    PUT /demo
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "products": {
      "properties": {
        "productName": {
          "type": "text",
          "analyzer": "ik_smart"
        },
        "size": {
          "type": "keyword"
        }
      }
    }
  }
}

    The command is successfully executed if the following information is displayed.

    {
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "demo"
}
  9. Log in to the CDM management console.
  10. Purchase a CDM cluster.

    For details, see Creating a Cluster in the Cloud Data Migration User Guide.

  11. Create a link between CDM and CSS.

    For details, see Creating a Link in the Cloud Data Migration User Guide.

  12. Create a link between CDM and OBS.

    For details, see Creating a Link in the Cloud Data Migration User Guide.

  13. Create a job on the purchased CDM cluster and migrate the data in the OBS bucket to the target cluster in CSS.

    For details, see Table/File Migration in the Cloud Data Migration User Guide.

  14. On the Console page of Kibana, search for the imported data.

    On the Console page of Kibana, run the following command to search for data. View the search results. If the searched data is consistent with the imported data, then the data has been imported successfully.

    GET demo/_search

    The command is successfully executed if the following information is displayed.

    {
  "took": 18,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 7,
    "max_score": 1,
    "hits": [
      {
        "_index": "demo",
        "_type": "products",
        "_id": "g6UepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"L"}""",
          "productName": """{"productName":"Latest art shirts for women in autumn 2017""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "hKUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"M"}""",
          "productName": """{"productName":"Latest art shirts for women in autumn 2017""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "haUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"S"}""",
          "productName": """{"productName":"Latest art shirts for women in autumn 2017""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "hqUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"M"}""",
          "productName": """{"productName":"Latest jeans for women in autumn 2018""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "h6UepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"S"}""",
          "productName": """{"productName":"Latest jeans for women in autumn 2018""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "iKUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"L"}""",
          "productName": """{"productName":"Latest casual pants for women in autumn 2017""""
        }
      },
      {
        "_index": "demo",
        "_type": "products",
        "_id": "iaUepnEBuvdFwWkRmn4V",
        "_score": 1,
        "_source": {
          "size": """"size":"S"}""",
          "productName": """{"productName":"Latest casual pants for women in autumn 2017""""
        }
      }
    ]
  }
}

    demo specifies the created index name. Set this parameter based on site requirements.