Error Occurs When MoXing Is Used to Copy Data

Symptom

1. When you call moxing.file.copy_parallel() to copy a file from the OBS bucket for a development environment to another bucket, the file is not visible in the target bucket.
2. An error occurs when MoXing is used to copy data. Example:
   • The following error occurs when MoXing is used to copy OBS data in the ModelArts development environment: keyError: 'request-id'
   • The error No files to copy occurs when ModelArts uses MoXing to copy data.
   • The error some parts are failed when download occurs when ModelArts uses MoXing to copy data.
   • socket.gaierror: [Errno -2] Name or service not known
   • ERROR:root:Failed to call:

     func=<bound method ObsClient.getObject of <obs.client.ObsClient object at 0x7fd705939710>>

     args=('bucket', 'data/TFRecord/HY_all_inside/no_adjust_light_3/09_06_6x128x128_0000000212.tfrecord')

   • TimeoutError: [Errno 110] Connection timed out
   • WARNING:root:Retry=9,Wait=0.1, Timestamp = 1567152567.5327423

Possible Cause

The possible causes are as follows:

• The source file does not exist.
• The target OBS path is incorrect or the two OBS paths are not in the same region.
• Space of the training job is insufficient.
• The training decryption component is abnormal.

Solution

Check the following items based on the error message:

1. Check whether the first parameter of moxing.file.copy_parallel() contains a file. If it contains no file, the error message "No files to copy" is displayed.
   • If the file exists, go to 2.
   • If the file does not exist, ignore the error and proceed with subsequent operations.
2. If a timeout error occurs, check whether the OBS path for copying data matches the region of your development environment or training job.

   Log in to the ModelArts management console, and view the region where ModelArts resides. Log in to OBS Console, and view the region where the OBS bucket resides. Check whether they are in the same region.

   • If they are in the same region, go to step 3.
   • If they are not in the same region, create a bucket and a folder in OBS that is in the same region as ModelArts, and upload data to the bucket.
3. If error code 403 occurs, check whether the OBS path is obs://xxx. You can check whether the OBS path exists as follows:

   mox.file.exists('obs://bucket_name/sub_dir_0/sub_dir_1')

   • If the path exists, go to 4.
   • If the path is not found, change it to an available OBS path.
4. If error some part are failed when download occurs, check whether CPU resources are used. The /cache directory of the CPUs and the code directory share 10 GB. The possible cause is insufficient space. Run this command in code to check the disk size:

   os.system('df -hT')

   • If disk space is sufficient, go to 5.
   • If disk space is insufficient, use GPU resources.
5. If data fails to be copied using MoXing in a notebook instance, run the df -hT command on the Terminal page to check the space size and check whether the failure cause is insufficient space. You can use EVS to attach disks when creating a notebook instance.
6. If error code 503 occurs or the request ID is empty, contact OBS engineers to monitor the bucket's backend and determine if traffic limits are due to full bandwidth or too many simultaneous requests. Adjust these settings as needed and retry.
7. If can't find xxx.temp or FileNotFoundError occurs, check whether multiple processes download the same file at the same time.