Remote SSH

Constraints

To connect to a training job instance using SSH, pay attention to the following constraints:

• The local user key and permission must match.
• The local user key must be stored in the specified directory.
  • Windows: C:\Users\{{user}}
  • macOS/Linux: ~
    

    On macOS and Linux, ~ indicates the home directory of the current user.

• ma-user in the remote image cannot be locked.
• Set the permission on the remote /home/ma-user/.ssh or /root/.ssh directory to 750 or 755.
• The OpenSSH version on the local or remote server cannot be earlier than 8.0.
• Do not create more than 10 connections at the same time.

If SSH connection problems occur, rectify the fault by referring to Accessing a Notebook Instance Using SSH for Troubleshooting.

Prerequisites

• A training job instance has been created with remote SSH enabled. The instance must be in the Running state.
  • For details about how to create a training job instance, see Creating a Custom Training Job.
  • For details about how to enable remote SSH development in an existing training job instance, see Modifying the SSH Configuration for a Notebook Instance.
• On the training job instance details page, find Remote SSH to obtain the development environment access address and port number.
• The key pair is available.

  A key pair is automatically downloaded after you create it. Securely store your key pair. If an existing key pair is lost, create a new one.

Remote SSH Development Address Description

The format of the remote SSH development address is as follows:

ssh -o StrictHostKeyChecking=no -i {key-file-name} {launch-user}@{domain-name} -p {port}

Assume that the remote SSH development address is as follows: The key file name is KeyPair-1234-test.pem, ma-user is the startup user, authoring-ssh-modelarts-*****.huawei.com is the domain name, and 31092 is the port number.

ssh -o StrictHostKeyChecking=no -i KeyPair-1234-test.pem ma-user@authoring-ssh-modelarts-*****.huawei.com -p 31092

Method 1: Using CMD to Connect to an On-Cloud Training Job Instance

The following uses Windows as an example.

1. Open CMD.
   • Option A: Using the start menu
     1. Click the start button in the lower-left corner of your screen, or press the Windows key on your keyboard.
     2. Type CMD in the start menu and click Command Prompt.
   • Option B: Using the run dialog box
     1. Press Windows + R on your keyboard to open the Run dialog box.
     2. Type CMD and press Enter or click OK.
2. In CMD, navigate to the directory where your key file is located and execute the following command to connect to the training instance:

   ssh -o StrictHostKeyChecking=no -i {key-file-name} {launch-user}@{domain-name} -p {port}

   Note: Replace {key-file-name}, {launch-user}, {domain-name}, and {port} with your actual information obtained from the Prerequisites.

   Example: If your key file name is KeyPair-1234-test.pem, and the remote SSH access address is ssh://ma-user@authoring-ssh-modelarts-*****.huawei.com:31092 (where ma-user is the launch user, authoring-ssh-modelarts-*****.huawei.com is the domain name, and 31092 is the port), the command will be:

   ssh -o StrictHostKeyChecking=no -i KeyPair-1234-test.pem ma-user@authoring-ssh-modelarts-*****.huawei.com -p 31092

   As shown in the figure below, the appearance of the interactive interface indicates a successful connection.

Method 2: Using MobaXterm to Connect to an On-Cloud Training Job Instance

1. Download and install the SSH remote access tool, for example, MobaXterm.
2. Open MobaXterm, click Session in the upper-left corner, and navigate to the SSH tab in the Session settings dialog box. Configure the Remote host, Username, Port, and Use private key, and click OK.
   Figure 1 Configuring a session
   

   Obtain the remote SSH access address and the key file path from the Prerequisites. For example, if the remote SSH access address is ssh://ma-user@authoring-ssh-modelarts-*****.huawei.com:31092, configure the settings as follows.

   Table 1 Parameters

   Configuration Item		Description	Example Value
   Basic SSH settings	Remote host	The address of the remote host, obtained from the remote SSH access address.	authoring-ssh-modelarts-*****.huawei.com
   	Username	The username used to connect to the remote host, obtained from the remote SSH access address.	ma-user
   	Port	The port number used to connect to the remote host, obtained from the remote SSH access address.	31092
   Advanced SSH settings	Use private key	The local file path of your private key, obtained from the Prerequisites.	C:\Users\***\KeyPair-1234-test.pem

   As shown in the figure below, the appearance of the interactive interface indicates a successful connection.

   Figure 2 Using MobaXterm to connect to a training instance
   

3. (Optional) Access other node containers in a distributed job.

In distributed training scenarios (specifically Volcano Jobs), a single job typically consists of multiple Tasks (e.g., worker-0, worker-1, etc.). The platform automatically configures SSH passwordless mutual trust between these instances, allowing you to jump from your currently logged-in node to any other node within the same job for seamless multi-node cluster debugging.

Procedure

1. Obtain the target node hostname. In your current container terminal, use the platform-injected environment variables to view the hostnames of all participating nodes:

   env | grep VC_%s_HOSTS

   Example:

   env | grep VC_WORKER_HOSTS

   Note: Volcano injects variables based on the Task name. For example:

   VC_WORKER_HOSTS=ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-0.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f,ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-1.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f

   If the number of nodes exceeds the threshold (500 nodes by default), variables can only be injected to worker-0. Example:

   VC_WORKER_HOSTS=ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-0.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f

2. Execute the SSH jump. Since passwordless mutual trust is pre-configured, you can jump to another worker node directly via the ssh command without entering a password:

   Note: The SSH port for switching between worker nodes is 2222. To switch to the worker-1 node, add port 2222 in the SSH command.

   # Jump to the worker-1 node

   ssh ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-1.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f -p 2222

3. Verify the jump result. Once the jump is successful, the terminal prompt will display the new hostname. You can confirm this by running the hostname command:

   hostname

   # Output example:

   ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f-worker-1.ma-job-40a6df3b-d136-452e-9b8c-530baf9b777f

Notes

• User consistency: The SSH session defaults to the currently logged-in user, ma-user.
• Environment variable scope: The target node also contains the platform-injected environment variables (e.g., VC_WORKER_HOSTS), allowing you to navigate freely between all nodes in the cluster.
• Mutual trust boundary: Passwordless login is restricted strictly to container instances within the same training job.