Help Center/ ModelArts/ DevEnviron/ JupyterLab/ Visualized Model Training/ TensorBoard Visualization Jobs
Updated on 2024-08-14 GMT+08:00
View PDF
Share

TensorBoard Visualization Jobs

ModelArts supports TensorBoard for visualizing training jobs. TensorBoard is a visualization tool package of TensorFlow. It provides visualization functions and tools required for machine learning experiments.

TensorBoard effectively displays the computational graph of TensorFlow in the running process, the trend of all metrics in time, and the data used in the training.

Prerequisites

When you write a training script, add the code for collecting the summary record to the script to ensure that the summary file is generated in the training result.

For details about how to add the code for collecting the summary record to a TensorFlow-powered training script, see TensorFlow official website.

Precautions

  • A running visualization job is not billed separately. When the target notebook instance is stopped, the billing stops.
  • If the summary file is stored in OBS, you will be charged for the storage. After a job is complete, stop the notebook instance and clear OBS data to stop billing.

Process of Creating a TensorBoard Visualization Job in a Development Environment

Step 1 Create a Development Environment and Access It Online

Step 2 Upload the Summary Data

Step 3 Start TensorBoard

Step 4 View Visualized Data on the Training Dashboard

Step 1 Create a Development Environment and Access It Online

On the ModelArts management console, choose Development Workspace > Notebook, and create an instance using a TensorFlow or PyTorch image. After the instance is created, click Open in the Operation column of the instance to access it online.

Only CPU and GPU flavors with TensorFlow2.1, PyTorch1.4, or PyTorch1.8 and later images can support TensorBoard visualization for training jobs. Select images and flavors based on the site requirements.

Step 2 Upload the Summary Data

Summary data is required for using TensorBoard visualization functions in DevEnviron.

You can upload the summary data to the /home/ma-user/work/ directory in the development environment or store it in the OBS parallel file system.

  • For details about how to upload the summary data to the notebook path /home/ma-user/work/, see Uploading Files to JupyterLab.
  • To store the summary data in an OBS parallel file system that is mounted to a notebook instance, upload the summary file generated during model training to the OBS parallel file system and ensure that the OBS parallel file system and ModelArts are in the same region. When TensorBoard is started in a notebook instance, the notebook instance automatically mounts the OBS parallel file system directory and reads the summary data.

Step 3 Start TensorBoard

There are multiple methods to open TensorBoard in JupyterLab in the development environment. Select one based on your habits.

Figure 1 Starting TensorBoard in JupyterLab

Method 1 (recommended):

  1. Open JupyterLab, in the navigation pane on the left, create the summary folder, and upload data to /home/ma-user/work/summary. The folder name must be summary.
  2. Go to the summary folder and click to go to the TensorBoard page. See Figure 2.
    Figure 2 TensorBoard page (1)

Method 2

You can upgrade TensorBoard to any version except 2.4.0. After the upgrade, the new version of TensorBoard is used only in method 2. For other methods, use TensorBoard 2.1.1.

  1. Click to go to the JupyterLab development environment. The .ipynb file is automatically created.
  2. Enter the following command in the dialog box: 
    %reload_ext ma_tensorboard
%ma_tensorboard  --port {PORT} --logdir {BASE_DIR}

    Parameters:

    • port {PORT}: web service port for visualization, which defaults to 8080. If the default port 8080 has been used, specify a port ranging from 1 to 65535.
    • logdir {BASE_DIR}: data storage path in the development environment
      • Local path of the development environment: ./work/xxx (relative path) or /home/ma-user/work/xxx (absolute path)
      • Path of the OBS parallel file system: obs://xxx/
    Example:
# If the summary data is stored in /home/ma-user/work/ of the development environment, run the following command:
%ma_tensorboard  --port {PORT} --logdir /home/ma-user/work/xxx 
or
# If the summary data is stored in the OBS parallel file system, run the following command and the development environment automatically mounts the storage path of the OBS parallel file system and reads data.
%ma_tensorboard  --port {PORT} --logdir obs://xxx/
    Figure 3 TensorBoard page (2)

Method 3

  1. Choose View > Activate Command Palette, enter TensorBoard in the search box, and click Create a new TensorBoard.
    Figure 4 Create a new TensorBoard
  2. Enter the path of the summary data you want to view or the storage path of the OBS parallel file system.
    • Local path of the development environment: ./summary (relative path) or /home/ma-user/work/summary (absolute path)
    • Path of the OBS parallel file system bucket: obs://xxx/
    Figure 5 Entering the summary data path
    Figure 6 TensorBoard page (3)

Method 4

Click and run the following command. The UI will not be displayed.

tensorboard --logdir ./log
Figure 7 Opening TensorBoard through Terminal

Step 4 View Visualized Data on the Training Dashboard

For TensorBoard visualization, you need the training dashboard. It lets you visualize scalars, images, and computational graphs.

For more functions, see Get started with TensorBoard.

Related Operations

To stop a TensorBoard instance, use any of the following methods:

  • Method 1: Click . The TensorBoard instance management page is displayed, which shows all started TensorBoard instances. Click SHUT DOWN next to an instance.
    Figure 8 Clicking SHUT DOWN to stop an instance
  • Method 2: Obtain PID by viewing on the startup screen or running the ps -ef | grep tensorboard command. Enter the following command in the .ipynb file window in JupyterLab: 
    !kill PID
  • Method 3: Click as shown in the following figure to stop all started TensorBoard instances.
    Figure 9 Stopping all started TensorBoard instances
  • (Not recommended) Method 4: Close the TensorBoard window in JupyterLab. This method closes only the window, but the instance is still running on the backend.