Obtaining CSV Files from OBS to Enrich Data

Prerequisites

CSV files have been uploaded to OBS.

Introduction

OBS is a highly reliable, high-performance, and high-security cloud service that provides secure and scalable object storage. For data that is not frequently updated, you are advised to store it on OBS with a small storage fee. Then you can obtain complete log data from OBS as required. LTS's data processing feature allows you to use the res_obs_file function to obtain data from OBS, use the tab_parse_csv function to build a table, and use the e_table_map function to match fields to return specified fields and field values, and generate new log data.

Use Cases

• Raw log

  {
      "account":"Sf24asc4ladDS"
  }

• CSV file data in OBS buckets

  Table 1 CSV file data

  id	account	nickname
  1	Sf24asc4ladDS	Doflamingo
  2	Sf24asc4ladSA	Kaido
  3	Sf24asc4ladCD	Roger

  

  CSV files must be in UNIX file format. Windows file format is not supported currently.

• Processing rule

  The account field in the raw logs is matched with that in OBS CSV files. The matching is successful only when the account field values are the same. After the matching is successful, the nickname field and its value in the OBS CSV file are returned, and the data is combined with the data in the raw logs to generate new data.

  e_table_map(tab_parse_csv(res_obs_file("https://obs.xxx.myhuaweicloud.com","dsl-test-xx","data.csv")),"account","nickname")

  The following table describes the key fields of the res_obs_file function. For more information, see Resource Functions.

  Field	Description
  endpoint	OBS access domain name. For details, see res_obs_file.
  bucket	OBS bucket for storing CSV files.
  file	Path of the target OBS file. For example, test/data.txt. It cannot start with a slash (/).

• Processing result

  {
  	"nickname": "Doflamingo",
  	"account": "Sf24asc4ladDS"
  }