Remote SSH

Constraints

To connect to a training job instance using SSH, pay attention to the following constraints:

• The local user key and permission must match.
• The local user key must be stored in the specified directory.
  • Windows: C:\Users\{{user}}
  • macOS/Linux: ~
    

    On macOS and Linux, ~ indicates the home directory of the current user.

• ma-user in the remote image cannot be locked.
• Set the permission on the remote /home/ma-user/.ssh or /root/.ssh directory to 750 or 755.
• The OpenSSH version on the local or remote server cannot be earlier than 8.0.
• Do not create more than 10 connections at the same time.

If the SSH connection is abnormal, rectify the fault by referring to Accessing a Notebook Instance Using SSH for Troubleshooting.

Prerequisites

• A training job instance has been created with remote SSH enabled. The instance must be in the Running state.
  • For details about how to create a training job instance, see the new training job creation page.
  • For details about how to enable remote SSH development in an existing training job instance, see Modifying the SSH Configuration for a Notebook Instance.
• The address and port number of the development environment are available. To obtain this information, go to the training job instance details page and get the Remote SSH infomation.
• The key pair is available.

  A key pair is automatically downloaded after you create it. Securely store your key pair. If an existing key pair is lost, create a new one.

Step 1 Installing the SSH Tool

Download and install the SSH remote access tool, for example, PuTTY.

Step 2 Using PuTTYgen to Convert the .pem Key Pair File to a .ppk Key Pair File

1. Download PuTTYgen and double-click it to run it.
2. Click Load to load the .pem key file created and saved during notebook instance creation.
3. Click Save private key to save the generated .ppk file. The file name can be customized, for example, key.ppk.
   Figure 1 Converting the .pem key pair file to a .ppk key pair file
   

Step 3 Using SSH to Connect to a Training Job Instance

1. Run PuTTY.
2. Click Session and set the following parameters:
   1. Host Name (or IP address): Address for accessing the on-cloud training job instance. Obtain the address on the page providing detailed information of the target training job instance.
   2. Port: Port number for accessing the on-cloud training job instance. Obtain the port number on the page providing detailed information of the target training job instance, for example, 32701.
   3. Connection type: Choose SSH.
   4. Saved Sessions: Task name, which can be clicked for remote access when you use PuTTY next time.
   Figure 2 Configuring Session
   
3. Choose Window > Translation and select UTF-8 from the drop-down list box in the Remote character set area.
   Figure 3 Setting the character format
   
4. Choose Connection > Data and enter ma-user for Auto-login username.
   Figure 4 Entering a username
   
5. Choose Connection > SSH > Auth, click Browse, and select the .ppk file generated in step 2.

   

6. Click Open. If you are logging in to the instance for the first time, PuTTY displays a security warning dialog box, asking if you want to accept the instance security certificate. Click Accept to save the certificate to your local registry.
   Figure 5 Asking if you want to accept the instance security certificate
   
7. Check that the on-cloud training job instance is connected.
   Figure 6 On-cloud training job instance connected
   

8. (Optional) Access other node containers in a distributed job.

In distributed training scenarios (specifically Volcano Jobs), a single job typically consists of multiple Tasks (e.g., worker-0, worker-1, etc.). The platform automatically configures SSH passwordless mutual trust between these instances, allowing you to jump from your currently logged-in node to any other node within the same job for seamless multi-node cluster debugging.

Procedure

1. Obtain the target node hostname. In your current container terminal, use the platform-injected environment variables to view the hostnames of all participating nodes:

   env | grep VC_%s_HOSTS

   Example:

   env | grep VC_WORKER_HOSTS

   Note: Volcano injects variables based on the Task name. For example:

   VC_WORKER_HOSTS=ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-0.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f,ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-1.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f

2. Execute the SSH jump. Since passwordless mutual trust is pre-configured, you can jump to another node directly via the ssh command without entering a password:

# Jump to the worker-1 node

ssh ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-1.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f

3. Verify the jump result. Once the jump is successful, the terminal prompt will display the new hostname. You can confirm this by running the hostname command:

hostname

# Output example: ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-1.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f

Notes

• User consistency: The SSH session defaults to the currently logged-in user, ma-user.
• Environment variable scope: The target node also contains the platform-injected environment variables (e.g., VC_WORKER_HOSTS), allowing you to navigate freely between all nodes in the cluster.
• Mutual trust boundary: Passwordless login is restricted strictly to container instances within the same training job.