Help Center> ModelArts> User Guide (Senior AI Engineers)> DevEnviron (Notebook)> Using Jupyter Notebook> Uploading and Downloading Data> Uploading Large Files to a Notebook Instance
View PDF

Uploading Large Files to a Notebook Instance

On the Notebook page, click Upload to upload a file. For details, see Uploading a File in Common Operations on Jupyter Notebook. If a message is displayed indicating that the size of the files to be uploaded exceeds the upper limit when uploading files to notebook instances or JupyterLab, you can upload the files to OBS and then download them to notebook instances.

Step 1: Uploading Files to OBS

You are advised to use an OBS tool, such as OBS Browser+ or obsutil, to upload large files because OBS Console has restrictions on the file size and quantity. OBS Browser+ is a graphical tool that provides complete functions for managing your buckets and objects in OBS. You are advised to use this tool to create buckets or upload objects. obsutil is a command line tool for accessing and managing OBS resources. If you are familiar with command line interface (CLI), obsutil is recommended as an ideal tool for batch processing and automated tasks. For details about how to upload files to OBS, see Object Storage Service Tools Guide.

You can upload files to buckets in different ways. Then, OBS stores the files in the buckets as objects.

Table 1 Uploading an object using different methods of access

Access Method

How to Upload an Object

Console

Uploading an object using OBS Console

OBS Browser+

Uploading an object using OBS Browser+

obsutil

Uploading an object using obsutil

SDK

Uploading an object using SDKs. For details, see the section about object uploading in the developer guide of each language.

API

Uploading objects - PUT and Uploading objects - POST

Step 2: Downloading Files from OBS to Notebook Instances

A notebook instance can be mounted to OBS or EVS as the storage location. The operation method varies depending on the instance types.

  • Downloading files to notebook instances with EVS attached
    • Use MoXing to synchronize files from OBS to notebook instances. For details, see Sample Code for Common Operations.
      Read an OBS file. For example, if you read the obs://bucket_name/obs_file.txt file, the content is returned as strings. 
      1
      		 
      file_str = mox.file.read('obs://bucket_name/obs_file.txt')
      You can also open the file object and read data from it. Both methods are equivalent. 
      1
2
      		 
      with mox.file.File('obs://bucket_name/obs_file.txt', 'r') as f:
  file_str = f.read()
    • Use the OBS API in the ModelArts SDK to download data from OBS to notebook instances.

      If the size of a single file exceeds 5 GB, the file cannot be uploaded in this mode. You are advised to use the MoXing API to upload large files.

      Sample code:

      1
2
3
      		 
      from modelarts.session import Session
session = Session()
session.download_data(bucket_path="/bucket-name/dir1/sdk.txt", path="/home/user/sdk/obs.txt")
  • Downloading files to notebook instances using OBS for data storage

    Upload files to the OBS path specified during notebook instance creation and synchronize the files from OBS to the notebook instances using Sync OBS.