Kuberay

vim my-rayjob.yaml

在使用Ray官方镜像时，可能由于网络原因拉取失败。建议将该镜像提前上传至SWR镜像仓库，并将文件内容中的镜像地址替换为SWR中的镜像地址。关于镜像上传SWR镜像仓库的详细步骤，请参见使用docker命令将镜像迁移至SWR。

文件内容如下：

apiVersion: ray.io/v1
kind: RayJob    # 资源类型为Ray Job
metadata:
  name: my-rayjob
spec:           # 定义Ray Job的规格，包括任务入口点、运行时环境和Ray Cluster配置
  entrypoint: python /home/ray/samples/sample_code.py     # 指定任务的入口点命令，这里是运行一个Python脚本
  runtimeEnvYAML: |
    env_vars:   # 定义任务运行时环境变量
      counter_name: "test_counter"
  rayClusterSpec:     # 用于定义Ray Cluster配置
    headGroupSpec:    # 定义Ray Cluster的Head节点配置
      rayStartParams: {}
      template:
        spec:
          containers:
            - name: ray-head
              image: rayproject/ray:2.41.0
              ports:
                - containerPort: 6379
                  name: gcs-server
                - containerPort: 8265 # Ray dashboard
                  name: dashboard
                - containerPort: 10001
                  name: client
              resources:
                limits:
                  cpu: "1"
                requests:
                  cpu: "200m"
              volumeMounts:
                - mountPath: /home/ray/samples
                  name: code-sample
          volumes:
            - name: code-sample
              configMap:
                name: ray-job-code-sample
                items:
                  - key: sample_code.py
                    path: sample_code.py
    workerGroupSpecs:    # 定义 Ray Cluster的Worker节点配置
      - replicas: 1
        minReplicas: 1
        maxReplicas: 5
        groupName: small-group
        rayStartParams: {}
        template:
          spec:
            containers:
              - name: ray-worker
                image: rayproject/ray:2.41.0
                resources:
                  limits:
                    cpu: "1"
                  requests:
                    cpu: "200m"
---
# 定义具体工作，里面包含一个Python脚本sample_code.py，该脚本实现了一个简单的分布式计数器
apiVersion: v1
kind: ConfigMap
metadata:
  name: ray-job-code-sample
data:
  sample_code.py: |
    import ray
    import os

    ray.init()

    @ray.remote
    class Counter:
        def __init__(self):
            # Used to verify runtimeEnv
            self.name = os.getenv("counter_name")
            assert self.name == "test_counter"
            self.counter = 0

        def inc(self):
            self.counter += 1

        def get_counter(self):
            return "{} got {}".format(self.name, self.counter)

    counter = Counter.remote()

    for _ in range(1000):
        ray.get(counter.inc.remote())
        print(ray.get(counter.get_counter.remote()))

kubectl create -f my-rayjob.yaml

回显如下，表示资源创建成功。

rayjob.ray.io/my-rayjob created
configmap/ray-job-code-sample created

1. 执行以下命令，检查Ray Cluster是否创建成功。

   kubectl get pod

   回显结果如下，则说明创建成功。其中，“my-rayjob-raycluster-4464z-head-hb5qx”表示Head Pod，“my-rayjob-raycluster-4464z-small-group-worker-csqb2”表示Worker Pod，“my-rayjob-x2tv6”用来查看作业的执行情况。如果“my-rayjob-x2tv6”的STATUS为Completed，则说明作业已完成。

   NAME                                                  READY    STATUS    RESTARTS    AGE
   my-rayjob-raycluster-4464z-head-hb5qx                 1/1      Running   0           24s
   my-rayjob-raycluster-4464z-small-group-worker-csqb2   1/1      Running   0           24s
   my-rayjob-x2tv6                                       1/1      Running   0           4s

2. 查看Ray Job的状态。

   kubectl get rayjob

   回显结果如下，则说明作业仍在进行中。如果JOB STATUS为SUCCEEDED，则说明作业已完成。

   NAME        JOB STATUS    DEPLOYMENT STATUS    RAY CLUSTER NAME              START TIME            END TIME    AGE
   my-rayjob   RUNNING       Running              my-rayjob-raycluster-4464z    2025-02-10T07:16:26Z              28

3. 执行以下命令，查看作业的具体执行情况。

   kubectl logs my-rayjob-x2tv6

   回显结果如下：

   2025-02-21 03:32:23,631 INFO cli.py:36 -- Job submission server address: http://my-rayjob-raycluster-4464z-head-svc.default.svc.cluster.local:8265
   2025-02-21 03:32:24,142 SUCC cli.py:60 -- --------------------------------------------
   2025-02-21 03:32:24,143 SUCC cli.py:61 -- Job 'my-rayjob-x2tv6' submitted successfully
   2025-02-21 03:32:24,143 SUCC cli.py:62 -- --------------------------------------------
   2025-02-21 03:32:24,143 INFO cli.py:286 -- Next steps
   2025-02-21 03:32:24,143 INFO cli.py:287 -- Query the logs of the job:
   2025-02-21 03:32:24,143 INFO cli.py:289 -- ray job logs my-rayjob-x2tv6
   2025-02-21 03:32:24,143 INFO cli.py:291 -- Query the status of the job:
   2025-02-21 03:32:24,143 INFO cli.py:293 -- ray job status my-rayjob-x2tv6
   2025-02-21 03:32:24,143 INFO cli.py:295 -- Request the job to be stopped:
   2025-02-21 03:32:24,143 INFO cli.py:297 -- ray job stop my-rayjob-x2tv6
   2025-02-21 03:32:24,147 INFO cli.py:304 -- Tailing logs until the job exits (disable with --no-wait):
   2025-02-21 03:32:25,011 INFO worker.py:1429 -- Using address 172.20.0.10:6379 set in the environment variable RAY_ADDRESS
   2025-02-21 03:32:25,012 INFO worker.py:1564 -- Connecting to existing Ray cluster at address: 172.20.0.10:6379...
   2025-02-21 03:32:25,017 INFO worker.py:1740 -- Connected to Ray cluster. View the dashboard at 172.20.0.10:8265 
   test_counter got 1
   test_counter got 2
   test_counter got 3
   test_counter got 4
   ...

1. 获取Ray Cluster中Head Pod的标签，使2中新建的Service能够识别并绑定Pod。请将“my-rayjob-raycluster-4464z-head-hb5qx”替换为您的Head Pod名称。

   kubectl describe pod my-rayjob-raycluster-4464z-head-hb5qx

   回显结果如下，建议使用“my-rayjob-raycluster-4464z-head”作为标签。

   ...
   Labels:           app.kubernetes.io/created-by=kuberay-operator
                     app.kubernetes.io/name=kuberay
                     ray.io/cluster=my-rayjob-raycluster-4464z
                     ray.io/group=headgroup
                     ray.io/identifier=my-rayjob-raycluster-4464z-head
                     ray.io/is-ray-node=yes
                     ray.io/node-type=head
   ...

2. 执行以下命令，创建YAML文件ray-dashboard.yaml，用于配置NodePort的Service，文件名称可自定义。该Service主要用于对外暴露服务，进而可以从浏览器直接访问Ray Dashboard。

   vim ray-dashboard.yaml

   文件内容如下：

   apiVersion: v1
   kind: Service
   metadata:
     name: ray-dashboard
     labels:
       ray.io/identifier: my-rayjob-raycluster-4464z-head
     namespace: default
   spec:
     ports:
     - name: cce-service-0
       port: 8265          # 访问Service的端口，设置为8265
       protocol: TCP       # 访问Service的协议，支持TCP和UDP
       targetPort: 8265    # Service访问目标容器的端口，此端口与容器中运行的应用强相关，必须为8265
     selector:             # 标签选择器
       ray.io/identifier: my-rayjob-raycluster-4464z-head
     externalTrafficPolicy: Cluster
     type: NodePort        # Service的类型，NodePort表示在通过节点端口访问

3. 执行以下命令，创建Service。

   kubectl create -f ray-dashboard.yaml

   回显结果如下，则说明Service创建成功。

   service/ray-dashboard created

4. 执行以下命令，查看该Service的节点端口信息。

   kubectl get services

   回显结果如下，其中32638即为节点端口。

   NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                AGE
   ray-dashboard                         NodePort    10.247.147.102   <none>        8265:32638/TCP                         8s

5. 在浏览器中输入“http://<集群任一节点的EIP>:32638”，访问Ray Dashboard页面。请确保集群中存在节点已绑定EIP，您可以在CCE控制台单击对应集群名称，在“节点管理 > 节点页面”，单击对应节点名称进入ECS页面绑定EIP，具体可参考绑定弹性公网IP。

   如果访问界面类似图3 Ray Dashboard，则说明访问成功。您可以根据Ray Dashboard实时监控和管理Ray Cluster的状态、资源使用情况和作业执行情况。例如，您可以在Recent jobs模块查看作业执行状况。

   图2 Ray Dashboard
   

1. 删除Ray Job相关资源。

   kubectl delete -f my-rayjob.yaml

   回显结果如下：

   rayjob.ray.io "my-rayjob" deleted
   configmap "ray-job-code-sample" deleted

2. 删除Service资源。

   kubectl delete -f ray-dashboard.yaml

   回显结果如下：

   service "ray-dashboard" deleted