Obtaining CSV Files from OBS to Enrich Data

Prerequisites

The CSV files have been uploaded to OBS.

Introduction

Huawei Cloud OBS is a highly reliable, high-performance, and secure object-based storage service. For data that is not frequently updated, OBS provides a cost-effective alternative for long-term retention, allowing you to retrieve complete log datasets whenever necessary. LTS's data processing feature allows you to use the res_obs_file function to pull data from OBS, the tab_parse_csv function to build a table, and the e_table_map function to match fields. This process returns designated fields and values to generate new log data.

Use Cases

• Raw log

  {
      "account":"Sf24asc4ladDS"
  }

• CSV file data in OBS buckets

  Table 1 CSV file data

  id	account	nickname
  1	Sf24asc4ladDS	Doflamingo
  2	Sf24asc4ladSA	Kaido
  3	Sf24asc4ladCD	Roger

  

  CSV files must be in UNIX format. Windows file format is not supported currently.

• Processing rule

  The account field in the raw logs is matched with that in the OBS CSV files. The matching is successful only when the account field values are the same. After the matching is successful, the nickname field and its value in the OBS CSV files are returned, and the data is combined with the data in the raw logs to generate new data.

  e_table_map(tab_parse_csv(res_obs_file("https://obs.xxx.myhuaweicloud.com","dsl-test-xx","data.csv")),"account","nickname")

  The following table describes the key fields of the res_obs_file function. For more information, see Resource Functions.

  Field	Description
  endpoint	OBS access domain name. For details, see res_obs_file.
  bucket	OBS bucket for storing CSV files.
  file	Path of the target OBS file. For example, test/data.txt. It cannot start with a slash (/).

• Processing result

  {
  	"nickname": "Doflamingo",
  	"account": "Sf24asc4ladDS"
  }