Help Center/ ModelArts/ MoXing Developer Guide/ Sample Code for Common Operations
Updated on 2024-09-30 GMT+08:00
View PDF
Share

Sample Code for Common Operations

Data Reads and Writes

  • Read an OBS file.
    For example, if you read the obs://bucket_name/obs_file.txt file, the content is returned as strings. 
    1
2
    		 
    import moxing as mox
file_str = mox.file.read('obs://bucket_name/obs_file.txt')
    You can also open the file object and read data from it. Both methods are the same. 
    1
2
3
    		 
    import moxing as mox
with mox.file.File('obs://bucket_name/obs_file.txt', 'r') as f:
  file_str = f.read()
  • Read a line from a file. A string that ends with a newline character is returned. You can also open the file object in OBS. 
    1
2
3
    		 
    import moxing as mox
with mox.file.File('obs://bucket_name/obs_file.txt', 'r') as f:
  file_line = f.readline()
  • Read all lines from a file. A list is returned, in which each element is a line and ends with a newline character. 
    1
2
3
    		 
    import moxing as mox
with mox.file.File('obs://bucket_name/obs_file.txt', 'r') as f:
  file_line_list = f.readlines()
  • Read an OBS file in binary mode.
    For example, if you read the obs://bucket_name/obs_file.bin file, the content is returned as bytes. 
    1
2
    		 
    import moxing as mox
file_bytes = mox.file.read('obs://bucket_name/obs_file.bin', binary=True)

    You can also open the file object and read data from it. Both methods are the same.

    1
2
3
    		 
    import moxing as mox
with mox.file.File('obs://bucket_name/obs_file.bin', 'rb') as f:
  file_bytes = f.read()

    One or all lines in a file opened in binary mode can be read with the same method.

  • Write a string to a file.
    For example, write Hello World! into the obs://bucket_name/obs_file.txt file. 
    1
2
    		 
    import moxing as mox
mox.file.write('obs://bucket_name/obs_file.txt', 'Hello World!')

    You can also open the file object and write data into it. Both methods are the same.

    1
2
3
    		 
    import moxing as mox
with mox.file.File('obs://bucket_name/obs_file.txt', 'w') as f:
  f.write('Hello World!')

    When you open a file in write mode or call mox.file.write, if the file to be written does not exist, the file will be created. If the file to be written already exists, the file is overwritten.

  • Append content to an OBS file.

    For example, append Hello World! to the obs://bucket_name/obs_file.txt file.

    1
2
    		 
    import moxing as mox
mox.file.append('obs://bucket_name/obs_file.txt', 'Hello World!')

    You can also open the file object and append content to it. Both methods are the same.

    1
2
3
    		 
    import moxing as mox
with mox.file.File('obs://bucket_name/obs_file.txt', 'a') as f:
  f.write('Hello World!')

    When you open a file in append mode or call mox.file.append, if the file to be appended does not exist, the file will be created. If the file to be appended already exists, the content is directly appended.

    If the size of the source file to be appended is large, for example, the obs://bucket_name/obs_file.txt file exceeds 5 MB, the append performance is low.

    If the file object is opened in write or append mode, when the write function is called, the content to be written is temporarily stored in the cache until the file object is closed (the file object is automatically closed when the with statement exits). Alternatively, you can call the close() or flush() function of the file object to write the file content.

List

  • List an OBS directory. Only the top-level result (relative path) is returned. Recursive listing is not performed.

    For example, if you list obs://bucket_name/object_dir, all files and folders in the directory are returned, but recursive queries are not performed.

    Assume that obs://bucket_name/object_dir is in the following structure:

    1
2
3
4
5
    		 
    bucket_name
      |- object_dir
        |- dir0
          |- file00
        |- file1

    Call the following code:

    1
2
    		 
    import moxing as mox
mox.file.list_directory('obs://bucket_name/object_dir')

    The following list is returned:

    ['dir0', 'file1']
  • Recursively list an OBS directory. All files and folders (relative paths) in the directory are returned, and recursive queries are performed.

    Assume that obs://bucket_name/object_dir is in the following structure:

    1
2
3
4
5
    		 
    bucket_name
      |- object_dir
        |- dir0
          |- file00
        |- file1

    Call the following code:

    1
2
    		 
    import moxing as mox
mox.file.list_directory('obs://bucket_name/object_dir', recursive=True)

    The following list is returned:

    ['dir0', 'dir0/file00', 'file1']

Create a Folder

Create an OBS directory, that is, an OBS folder. Recursive creation is supported. That is, if the sub_dir_0 folder does not exist, it is automatically created. If the sub_dir_0 folder exists, no folder will be created.

1
2
 
import moxing as mox
mox.file.make_dirs('obs://bucket_name/sub_dir_0/sub_dir_1')

Query

  • Check whether an OBS file exists. If the file exists, True is returned. If the file does not exist, False is returned. 
    1
2
    		 
    import moxing as mox
mox.file.exists('obs://bucket_name/sub_dir_0/file.txt')
  • Check whether an OBS folder exists. If the folder exists, True is returned. If the folder does not exist, False is returned. 
    1
2
    		 
    import moxing as mox
mox.file.exists('obs://bucket_name/sub_dir_0/sub_dir_1')

    OBS allows files and folders with the same name exist (not allowed in UNIX). If a file or folder with the same name exists, for example, obs://bucket_name/sub_dir_0/abc, when mox.file.exists is called, True is returned regardless of whether abc is a file or folder.

  • Check whether an OBS path is a folder. If it is a folder, True is returned. If it is not a folder, False is returned. 
    1
2
    		 
    import moxing as mox
mox.file.is_directory('obs://bucket_name/sub_dir_0/sub_dir_1')

    OBS allows files and folders with the same name exist (not allowed in UNIX). If a file or folder with the same name exists, for example, obs://bucket_name/sub_dir_0/abc, when mox.file.is_directory is called, True is returned.

  • Obtain the size of an OBS file, in bytes.
    For example, obtain the size of obs://bucket_name/obs_file.txt. 
    1
2
    		 
    import moxing as mox
mox.file.get_size('obs://bucket_name/obs_file.txt')
  • Recursively obtain the size of all files in an OBS folder, in bytes.
    For example, obtain the total size of all files in the obs://bucket_name/object_dir directory. 
    1
2
    		 
    import moxing as mox
mox.file.get_size('obs://bucket_name/object_dir', recursive=True)
  • Obtain the stat information about an OBS file or folder. The stat information contains the following:
    • length: file size
    • mtime_nsec: creation timestamp
    • is_directory: whether the path is a folder
    For example, if you want to query the OBS file obs://bucket_name/obs_file.txt, you can replace the file path with a folder path. 
    1
2
3
4
5
    		 
    import moxing as mox
stat = mox.file.stat('obs://bucket_name/obs_file.txt')
print(stat.length)
print(stat.mtime_nsec)
print(stat.is_directory)

Delete

  • Delete an OBS file.
    For example, delete obs://bucket_name/obs_file.txt. 
    1
2
    		 
    import moxing as mox
mox.file.remove('obs://bucket_name/obs_file.txt')
  • Delete an OBS folder and recursively delete all content in the folder. If the folder does not exist, an error is reported.
    For example, delete all content in obs://bucket_name/sub_dir_0. 
    1
2
    		 
    import moxing as mox
mox.file.remove('obs://bucket_name/sub_dir_0', recursive=True)

Move and Copy

  • Move an OBS file or folder. The move operation is implemented by copying and deleting data.
    • Move an OBS file to another OBS file. For example, move obs://bucket_name/obs_file.txt to obs://bucket_name/obs_file_2.txt. 
      1
2
      		 
      import moxing as mox
mox.file.rename('obs://bucket_name/obs_file.txt', 'obs://bucket_name/obs_file_2.txt')

      The move and copy operation must be performed in the same bucket.

    • Move an OBS file to a local file. For example, move obs://bucket_name/obs_file.txt to /tmp/obs_file.txt. 
      1
2
      		 
      import moxing as mox
mox.file.rename('obs://bucket_name/obs_file.txt', '/tmp/obs_file.txt')
    • Move a local file to an OBS file. For example, move /tmp/obs_file.txt to obs://bucket_name/obs_file.txt. 
      1
2
      		 
      import moxing as mox
mox.file.rename('/tmp/obs_file.txt', 'obs://bucket_name/obs_file.txt')
    • Move a local file to another local file. For example, move /tmp/obs_file.txt to /tmp/obs_file_2.txt. This operation is equivalent to os.rename. 
      1
2
      		 
      import moxing as mox
mox.file.rename('/tmp/obs_file.txt', '/tmp/obs_file_2.txt')

    You can move folders in the same way. If you move a folder, all content in the folder is moved recursively.

  • Copy a file. mox.file.copy can be used to perform operations only on files. To perform operations on folders, use mox.file.copy_parallel.
    • Copy an OBS file to another OBS path. For example, copy obs://bucket_name/obs_file.txt to obs://bucket_name/obs_file_2.txt. 
      1
2
      		 
      import moxing as mox
mox.file.copy('obs://bucket_name/obs_file.txt', 'obs://bucket_name/obs_file_2.txt')
    • Copy an OBS file to a local path, that is, download an OBS file. For example, download obs://bucket_name/obs_file.txt to /tmp/obs_file.txt. 
      1
2
      		 
      import moxing as mox
mox.file.copy('obs://bucket_name/obs_file.txt', '/tmp/obs_file.txt')
    • Copy a local file to OBS, that is, upload an OBS file. For example, upload /tmp/obs_file.txt to obs://bucket_name/obs_file.txt. 
      1
2
      		 
      import moxing as mox
mox.file.copy('/tmp/obs_file.txt', 'obs://bucket_name/obs_file.txt')
    • Copy a local file to another local path. This operation is equivalent to shutil.copyfile. For example, copy /tmp/obs_file.txt to /tmp/obs_file_2.txt. 
      1
2
      		 
      import moxing as mox
mox.file.copy('/tmp/obs_file.txt', '/tmp/obs_file_2.txt')
  • Copy a folder. mox.file.copy_parallel can be used to perform operations only on folders. To perform operations on files, use mox.file.copy.
    • Copy an OBS file to another OBS path. For example, copy obs://bucket_name/sub_dir_0 to obs://bucket_name/sub_dir_1. 
      1
2
      		 
      import moxing as mox
mox.file.copy_parallel('obs://bucket_name/sub_dir_0', 'obs://bucket_name/sub_dir_1')
    • Copy an OBS folder to a local path, that is, download an OBS folder. For example, download obs://bucket_name/sub_dir_0 to /tmp/sub_dir_0. 
      1
2
      		 
      import moxing as mox
mox.file.copy_parallel('obs://bucket_name/sub_dir_0', '/tmp/sub_dir_0')
    • Copy a local folder to OBS, that is, upload an OBS folder. For example, upload /tmp/sub_dir_0 to obs://bucket_name/sub_dir_0. 
      1
2
      		 
      import moxing as mox
mox.file.copy_parallel('/tmp/sub_dir_0', 'obs://bucket_name/sub_dir_0')
    • Copy a local folder to another local path. This operation is equivalent to shutil.copytree. For example, copy /tmp/sub_dir_0 to /tmp/sub_dir_1. 
      1
2
      		 
      import moxing as mox
mox.file.copy_parallel('/tmp/sub_dir_0', '/tmp/sub_dir_1')