Notebook

Fundamental Concepts

Understanding the following core cloud-native concepts before using this add-on will help you manage your development environment more effectively:

• Notebook Custom Resource Definition (CRD): the core abstraction introduced by this add-on. The YAML file with kind: Notebook is a declarative configuration. You simply specify your requirements, and the controller automatically manages all underlying details, such as creating pods, mounting volumes, and configuring networks.
• PersistentVolumeClaim (PVC): In cloud-native environments, containers are ephemeral by design. Data in a pod is lost on pod restart. A PVC functions like a cloud USB drive mounted to a pod. It persistently stores your code, Conda environments, and personal configuration files, decoupling storage from compute.
• StatefulSet: the underlying workload controller that manages notebook pods. Unlike Deployments, StatefulSets maintain stable network identities and ensure your PVC is accurately remounted after rescheduling (for example, during a node failover).
• Resource culling: a cost-optimization mechanism. The controller monitors environment activity (such as Jupyter kernel or terminal connection statuses). When an environment has been idle for an extended period, the controller automatically hibernates it by scaling pods to zero, releasing compute resources back to the cluster while preserving all persistent data.

Prerequisites

• A CCE standard or Turbo cluster v1.29 or later is available. For details, see Buying a Standard/Turbo Cluster.
• Target service nodes and add-ons are provisioned with adequate resources.
  • NPU services: NPU nodes are created with CCE AI Suite (Ascend NPU) installed.
  • GPU services: GPU nodes are created with CCE AI Suite (NVIDIA GPU) installed.

Precautions

Uninstalling Notebook does not automatically delete its CRDs or CRs. Delete them manually to complete cleanup.

Notes and Constraints

• This add-on only deploys the notebook-controller for Notebook pods and manages their configurations. Notebook images must be custom-built or sourced from the community.

• This add-on is in initial rollout. Check the console for supported regions.
• This add-on is in OBT. You can access the latest features, but stability is not fully verified and no CCE SLA applies.

Billing

To access Notebook pods, bind an EIP to any cluster node. EIP usage is billed. For details, see EIP Price Calculator.

Installing the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Add-ons. Locate Notebook on the right and click Install.
3. In the Install Add-on sliding window, configure the specifications.

   Parameter	Example Value	Description
   Pods	1	At least one notebook-controller pod will be created when this add-on is installed.
   CPU Quota	Request: 100m	The minimum CPU request per container.
   	Limit: 500m	The maximum CPU available to each container. If the CPU usage exceeds this limit, the pod may be restarted, affecting add-on availability.
   Memory Quota	Request: 100 MiB	The minimum memory request per container.
   	Limit: 500 MiB	The maximum memory available to each container. If the memory usage exceeds this limit, the pod may be restarted, affecting add-on availability.

4. In the Parameters area, properly configure the parameters below based on applications.

   Parameter	Example Value	Description
   clusterDomain	cluster.local	The DNS domain suffix for internal Kubernetes cluster services. The controller uses it to generate fully-qualified domain names (FQDNs) for Notebook Services, for example, notebook-name.namespace.svc.cluster.local. This parameter defaults to cluster.local. If you customized the internal domain name during CCE cluster creation, update this parameter accordingly. A mismatch will cause Service communication failures.
   enableCulling	false	Whether to enable automatic reclamation of idle Notebook pods. true: enabled. The controller periodically identifies inactive Notebook pods and scales them in to zero, freeing underlying compute resources. Data on attached disks is preserved. false: disabled. Once started, pods continue running until manually deleted or scaled to zero. NOTE: Enable this function (true) in production environments with high-cost compute resources such as GPUs or NPUs to significantly reduce idle resource expenditure.
   cullIdleTime	1440	The idle timeout (in minutes) before automatic reclamation of Notebook pods. This specifies the maximum duration a Notebook pod can remain active without any activity, such as no active connections or code execution. The example value 1440 represents 24 hours. If a Notebook pod is idle for 24 consecutive hours, CCE automatically suspends it. You can reduce this duration based on your organization's resource needs, for example, set it to 120 (2 hours) or 240 (4 hours). CAUTION: This parameter applies only when enableCulling is set to true.
   idlenessCheckPeriod	1	The polling interval (in minutes) for detecting idle status. This specifies how often the controller sends a detection request to a Notebook pod's active status API to monitor HTTP traffic and Jupyter kernel status. The example value 1 indicates that the idle time is detected and calculated every minute. NOTE: Retain the default setting in most cases. If your cluster is large (thousands of pods), you can increase this value, for example, set it to 5, to reduce query pressure on the Kubernetes API server and controller.

5. Click Install.

Components

Notebook CR Settings

The Notebook CRD acts as a lightweight controller shell. It maps the spec.template.spec field to a complete, native Kubernetes PodSpec. Consequently, Notebook pods natively support all advanced capabilities provided by the underlying Kubernetes system.

• Common Notebook CR settings

  apiVersion: kubeflow.org/v1
  kind: Notebook
  metadata:
    name: advanced-ai-workspace
    namespace: team-algorithm
    labels:
      project: llm-pretraining
    annotations:
      # [Dedicated configuration] Allow the frontend dashboard to display custom descriptions.
      notebooks.kubeflow.org/title: "LLM Core Development Environment"
      # [Advanced scheduling] Use Volcano Scheduler for optimized NPU/GPU topology.
      volcano.sh/scheduler-name: volcano
  spec:
    template:
      spec:
        # [Security policy] Specify that all processes within the container run as a non-root user by default (UID 1000, typically jovyan).
        securityContext:
          runAsUser: 1000
          fsGroup: 100
  
        # [Node scheduling] Schedule pods only to heterogeneous compute node pools with specific labels.
        nodeSelector:
          node.kubernetes.io/instance-type: "accelerated-compute-pool"
  
  
        # [Taint tolerance] Tolerate taints on AI-dedicated nodes to prevent general service pods from preempting AI workload resources.
        tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
  
  
        containers:
        - name: notebook
          image: custom-registry.io/jupyter-ai-base:v2.0
          imagePullPolicy: Always
  
          # [Environment variables] Inject general environment configurations or service parameters.
          env:
          - name: JUPYTER_ENABLE_LAB
            value: "yes"
          - name: MODEL_CACHE_DIR
            value: "/data/models"
  
  
          # [Resource declaration] Declare core compute resource configurations.
          resources:
            requests:
              cpu: "8"
              memory: "32Gi"
            limits:
              cpu: "8"
              memory: "32Gi"
              # [Heterogeneous computing] Use accelerator cards exclusively. Specify GPU or NPU identifiers as needed.
              nvidia.com/gpu: 1           
              # huawei.com/Ascend910: 1   
  
          volumeMounts:
          - mountPath: /home/jovyan/work       # Personal workspace (read/write)
            name: workspace-pvc
          - mountPath: /data/public-datasets   # Public datasets (read-only)
            name: dataset-pvc
          - mountPath: /dev/shm                # Shared memory optimization for multi-process data loading bottlenecks
            name: dshm
  
  
        volumes:
        # 1. Mount a persistent cloud disk for code storage.
        - name: workspace-pvc
          persistentVolumeClaim:
            claimName: workspace-claim-user01
        # 2. Mount a high-performance file system (for example, SFS Turbo) for large datasets.
        - name: dataset-pvc
          persistentVolumeClaim:
            claimName: public-datasets-claim
        # 3. Mount a memory file system to accelerate multi-process reads (for example, PyTorch DataLoader).
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "8Gi"

• Suggestions for production environments
  • Stateless images with stateful storage

    Do not install time-consuming underlying dependencies (such as a specific CUDA/CANN version or C++ libraries requiring large-scale compilation) directly in a Notebook instance using pip install or apt-get. These installations are ephemeral and will be lost when the instance is reclaimed or restarted, making environments non-reproducible and forcing repeated rebuilds. Recommended approach:

    • Build Docker base images containing the underlying drivers and deep learning frameworks (such as PyTorch or MindSpore) to standardize and reuse environments.
    • Mount persistent storage to /home/jovyan/work and use venv or Conda for persistent installation and version management of lightweight Python dependencies (such as the requests library).
  • Unified network and route naming

    When a large number of Notebook instances are created concurrently, a large number of Services with identical names are generated in the cluster, causing network conflicts, resource contention, and management difficulties. To prevent these issues, follow these rules:

    • Each service team must have an independent Kubernetes namespace to isolate environments.
    • Notebook names must follow the format [Username]-[Project abbreviation]-[Date] to facilitate billing audits and batch identification and cleanup of orphaned instances.

Use Cases

This example demonstrates how to run CUDA code on a GPU-enabled Notebook instance (using a P4 GPU). Ensure the cluster contains GPU nodes and has the GPU and Notebook add-ons installed.

1. Create a Notebook instance.
   1. Install kubectl on an existing ECS and use kubectl to access the target cluster. For details, see Accessing a Cluster Using kubectl.
   2. Create a YAML file for pods configured with virtual GPU burst scheduling.

      vim notebook-sample.yaml

      The file content is as follows:

      apiVersion: kubeflow.org/v1
      kind: Notebook
      metadata:
        name: notebook-sample-v1    # Name of the Notebook instance
      spec:
        template:
          spec:
            containers:
              - name: notebook-sample-v1     # Name of the container
                image: quay.io/jupyter/pytorch-notebook:cuda13-latest  # Path to the used Notebook image
                resources:
                  limits:
                    cpu: 250m
                    memory: 512Mi
                    nvidia.com/gpu: '1'
                  requests:
                    cpu: 250m
                    memory: 512Mi
                    nvidia.com/gpu: '1'

      In this example, the image quay.io/jupyter/pytorch-notebook:cuda13-latest is hosted in an external Docker registry. Ensure your CCE nodes can access this registry. If your nodes cannot reach the registry, pull the image and upload it to Huawei Cloud SWR beforehand.

   3. Create the Notebook instance.

      kubectl apply -f notebook-sample.yaml

      If information similar to the following is displayed, the Notebook has been created:

      notebook.kubeflow.org/notebook-sample-v1 created

   4. Check the Notebook CR status.

      kubectl get Notebook

      Information similar to the following is displayed:

      NAME                 AGE
      notebook-sample-v1   25h

      If your Notebook name (for example, notebook-sample-v1) is displayed in the list, the CR has been created.

   5. View the created pod.

      kubectl get pod |grep notebook-sample-v1

      Information similar to the following is displayed:

      notebook-sample-v1-0             1/1     Running   0               28m

      If the command output contains Running, the instance is running.

2. Access the Notebook.
   1. View the label of the Notebook pod.

      kubectl describe pod notebook-sample-v1-0

      Key output:

      Labels:           apps.kubernetes.io/pod-index=0
                        controller-revision-hash=notebook-sample-v1-6bb8bb48ff
                        notebook-name=notebook-sample-v1
                        statefulset=notebook-sample-v1
                        statefulset.kubernetes.io/pod-name=notebook-sample-v1-0

      

      Use statefulset=notebook-sample-v1 as the label, as it is a native, stable, and predictable Kubernetes management label.

   2. Create a Service.
      1. Create notebook-service.yaml (filename customizable) to configure a NodePort Service. This Service exposes the Notebook so it can be accessed directly from a browser.

         vim notebook-service.yaml

         File content:

         apiVersion: v1
         kind: Service
         metadata:
           name: notebook-sample-v1
           namespace: default
         spec:
           externalTrafficPolicy: Cluster
           internalTrafficPolicy: Cluster
           ports:
           - name: http-notebook-sample-v1
             port: 80                # Port exposed by the Service
             protocol: TCP
             targetPort: 8888  # Port on the target container. This corresponds to the application port (typically 8888 for Jupyter Notebook).
           selector:
             statefulset: notebook-sample-v1
           sessionAffinity: None
           type: NodePort  # Service type. NodePort exposes the Service on each node's IP at a fixed port.

      2. Create the Service.

         kubectl create -f notebook-service.yaml

         If the command output contains created, the Service has been created.

      3. Check the Service NodePort.

         kubectl get services

         In the command output shown below, 32666 is the assigned NodePort.

         NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
         notebook-sample-v1   NodePort    10.247.**.**     <none>        80:32666/TCP   24h

   3. Obtain a token.
      1. Enter the Notebook container.

         kubectl exec -it notebook-sample-v1-0 -- bash

      2. Obtain the token.

         jupyter server list

         Command output:

         Currently running servers:
         http://localhost:8888/?token=ab23a4dfd63b... :: /home/jovyan

   4. Access the Jupyter UI. In the browser address bar, enter http://<EIP_of_any_node>:32666 to open the Notebook.

      Ensure an EIP is bound to a cluster node. Log in to the CCE console, click the cluster name to access the cluster console, choose Nodes in the navigation pane, click the Nodes tab in the right pane, and click the node name to go to the ECS console. Then, bind an EIP to the target ECS. For details, see Binding an EIP.

   5. On the login page, enter the obtained token and click Log in.

      

3. Test connectivity.
   1. On the Jupyter page, click Python 3.

      

   2. Run the following script:

      import torch
      
      # 1. Check if GPU is available
      print("1. Is GPU available:", torch.cuda.is_available())
      
      if torch.cuda.is_available():
          # 2. Try creating a simple Tensor in GPU memory
          x = torch.tensor([1.0, 2.0], device='cuda')
          print("2. Tensor located on device:", x.device)
      else:
          print("No GPU detected, please check your environment configuration.")

      Information similar to the following is displayed:

      Is GPU available: True
      Tensor located on device: cuda:0

      

FAQs

• How to Obtain and Select a Proper Notebook Image?

  A container image determines the software, frameworks, and underlying drivers pre-installed in the development environment. To ensure successful startup, the image must comply with basic containerized interface standards. The following describes common methods for obtaining high-quality images:

  • Kubeflow official community images (general-purpose CPU/GPU)

    The Kubeflow community maintains a series of rigorously tested base images covering the most commonly used data science stacks. You can pull these directly from the public image repository:

    • Basic data science (including Pandas and Scikit-learn): kubeflow/jupyter-scipy:latest
    • Official PyTorch integration: kubeflow/jupyter-pytorch:latest
    • Official TensorFlow integration: kubeflow/jupyter-tensorflow:latest

      How to obtain: Search for the image with a specific tag on Docker Hub (hub.docker.com/u/kubeflow).

  • Jupyter Docker Stacks (upstream open-source images)
    • Versatile data science stack: jupyter/datascience-notebook

      How to obtain: Visit Selecting an Image in the Jupyter official documentation, or search Docker Hub under the jupyter namespace.

Release History