Help Center > > User Guide> MRS Quick Start> Managing Files

Managing Files

Updated at: Mar 17, 2020 GMT+08:00

You can create directories, delete directories, and import, export, or delete files on the Files tab page in an analysis cluster with Kerberos authentication disabled.

Background

Data to be processed by MRS is stored in either OBS or HDFS. OBS provides you with massive, highly reliable, and secure data storage capabilities at a low cost. You can view, manage, and use data through OBS Console or OBS Browser.

Importing Data

MRS supports data import from the OBS system to HDFS. This function is recommended if the data size is small, because the upload speed reduces as the file size increases.

Both files and folders containing files can be imported. The operations are as follows:

  1. Log in to the MRS management console.
  2. Choose Clusters > Active Clusters, select a cluster, and click its name to switch to the cluster details page.
  3. Click Files to go to the Files tab page.
  4. Select HDFS File List.
  5. Click the data storage directory, for example, bd_app1.

    bd_app1 is just an example. The storage directory can be any directory on the page. You can create a directory by clicking Create Folder.

  6. Click Import Data and configure the paths for HDFS and OBS.

    When configuring the OBS or HDFS path, click Browse, select the file path, and click OK.

    • The path for OBS
      • It must start with s3a://.
      • Files and programs encrypted by the KMS cannot be imported.
      • Empty folders cannot be imported.
      • Directories and file names can contain letters, Chinese characters, digits, hyphens (-), or underscores (_), but cannot contain special characters (;|&><'$*?\).
      • Directories and file names cannot start or end with spaces, but can have spaces between other characters.
      • The full path of OBS contains a maximum of 255 characters.
    • The path for HDFS
      • It starts with /user by default.
      • Directories and file names can contain letters, Chinese characters, digits, hyphens (-), or underscores (_), but cannot contain special characters (;|&><'$*?\).
      • Directories and file names cannot start or end with spaces, but can have spaces between other characters.
      • The full path of HDFS contains a maximum of 255 characters.
  7. Click OK.

    View the upload progress in File Operation Records. The data import operation is operated as a Distcp job by MRS. You can check whether the Distcp job is successfully executed on the Jobs tab page.

Exporting Data

After data is processed and analyzed, you can either store the data in HDFS or export it to the OBS system.

Both files and folders containing files can be exported. The operations are as follows:

  1. Log in to the MRS management console.
  2. Choose Clusters > Active Clusters, select a cluster, and click its name to switch to the cluster details page.
  3. Click Files to go to the Files tab page.
  4. Select HDFS File List.
  5. Click the data storage directory, for example, bd_app1.
  6. Click Export Data and configure the paths for HDFS and OBS.

    When configuring the OBS or HDFS path, click Browse, select the file path, and click OK.

    • The path for OBS
      • It must start with s3a://.
      • Directories and file names can contain letters, Chinese characters, digits, hyphens (-), or underscores (_), but cannot contain special characters (;|&><'$*?\).
      • Directories and file names cannot start or end with spaces, but can have spaces between other characters.
      • The full path of OBS contains a maximum of 255 characters.
    • The path for HDFS
      • It starts with /user by default.
      • Directories and file names can contain letters, Chinese characters, digits, hyphens (-), or underscores (_), but cannot contain special characters (;|&><'$*?\).
      • Directories and file names cannot start or end with spaces, but can have spaces between other characters.
      • The full path of HDFS contains a maximum of 255 characters.

    Ensure that the exported folder is not empty. If an empty folder is exported to the OBS system, the folder is exported as a file. After the folder is exported, its name is changed, for example, from test to test-$folder$, and its type is file.

  7. Click OK.

    View the upload progress in File Operation Records. The data export operation is operated as a Distcp job by MRS. You can check whether the Distcp job is successfully executed on the Jobs tab page.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel