更新时间:2026-11-02 GMT+08:00
分享

starVLA模型

Notebook创建成功后,需要进入内置算法目录、激活conda环境,并下载所需的模型和数据,为后续的模型二次开发训练做好准备。

进入内置算法目录并激活环境

  1. 通过JupyterLab或SSH工具接入Notebook实例,打开Terminal终端。
  2. 进入内置算法目录。

    cd /opt/cloud/starvla_ascend_notebook/starVLA

  3. 激活名为“starvla”的conda虚拟环境。

    conda activate starvla

    激活成功后,终端命令行前缀会显示“(starvla)”,表示已进入starVLA算法的Python虚拟环境。

下载基模型和数据

通过huggingface-cli工具从huggingface网站下载所需的预训练模型和LeRobot v3.0格式的libero数据,也可以从CloudRobo平台下载v3格式的数据。

  1. 执行以下命令下载基模型到本地目录。

    hf download Qwen/Qwen3-VL-4B-Instruct --local-dir ./playground/Pretrained_models/Qwen3-VL-4B-Instruct
    hf download StarVLA/Qwen3-VL-4B-Instruct-Action --local-dir ./playground/Pretrained_models/Qwen3-VL-4B-Instruct-Action
    hf download physical-intelligence/fast --local-dir ./playground/Pretrained_models/fast

  2. 执行以下命令下载数据到本地目录。

    • 从huggingface下载LeRobot v3.0格式的libero数据。
      hf download lerobot/libero --repo-type dataset --local-dir ./playground/Dataset/libero
    • 从具身广场下载v3格式的数据。

      先配置访问密钥,然后才能下载v3格式数据。

配置数据

以LeRobot v3.0格式的libero数据为例来说明完整的配置流程。

  1. 在数据的meta目录下添加modality.json。

    {
        "state": {
            "x": {
                "start": 0,
                "end": 1
            },
            "y": {
                "start": 1,
                "end": 2
            },
            "z": {
                "start": 2,
                "end": 3
            },
            "roll": {
                "start": 3,
                "end": 4
            },
            "pitch": {
                "start": 4,
                "end": 5
            },
            "yaw": {
                "start": 5,
                "end": 6
            },
            "pad": {
                "start": 6,
                "end": 7
            },
            "gripper": {
                "start": 7,
                "end": 8
            }
        },
        "action": {
            "x": {
                "start": 0,
                "end": 1
            },
            "y": {
                "start": 1,
                "end": 2
            },
            "z": {
                "start": 2,
                "end": 3
            },
            "roll": {
                "start": 3,
                "end": 4
            },
            "pitch": {
                "start": 4,
                "end": 5
            },
            "yaw": {
                "start": 5,
                "end": 6
            },
            "gripper": {
                "start": 6,
                "end": 7
            }
        },
        "video": {
            "primary_image": {
                "original_key": "observation.images.image"
            },
            "wrist_image": {
                "original_key": "observation.images.image2"
            }
        },
        "annotation": {
            "human.action.task_description": {
                "original_key": "task_index"
            }
        }
    }

  2. 若采用datasets.vla_data.data_mix对数据进行混合采样,训练前需要先将数据注册到代码里。

    1. 在./starVLA/dataloader/gr00t_lerobot/data_config.py中新建DataConfig类。

    class Libero4in1DataConfig:
        embodiment_tag = EmbodimentTag.FRANKA
        video_keys = [
            "video.primary_image",
            "video.wrist_image",
        ]
    
        state_keys = [
            "state.x",
            "state.y",
            "state.z",
            "state.roll",
            "state.pitch",
            "state.yaw",
            "state.pad",
            "state.gripper",
        ]
        action_keys = [
            "action.x",
            "action.y",
            "action.z",
            "action.roll",
            "action.pitch",
            "action.yaw",
            "action.gripper",
        ]
    
        language_keys = ["annotation.human.action.task_description"]
    
        observation_indices = [0]
        action_indices = list(range(8))
        state_indices = list(range(-16, 0))
    
        def modality_config(self):
            video_modality = ModalityConfig(
                delta_indices=self.observation_indices,
                modality_keys=self.video_keys,
            )
            state_modality = ModalityConfig(
                delta_indices=self.state_indices,
                modality_keys=self.state_keys,
            )
            action_modality = ModalityConfig(
                delta_indices=self.action_indices,
                modality_keys=self.action_keys,
            )
            language_modality = ModalityConfig(
                delta_indices=self.observation_indices,
                modality_keys=self.language_keys,
            )
            modality_configs = {
                "video": video_modality,
                "state": state_modality,
                "action": action_modality,
                "language": language_modality,
            }
            return modality_configs
    
        def transform(self):
            transforms = [
                # action transforms
                StateActionToTensor(apply_to=self.action_keys),
                StateActionTransform(
                apply_to=self.action_keys,
                normalization_modes={
                    "action.x": "min_max",
                    "action.y": "min_max",
                    "action.z": "min_max",
                    "action.roll": "min_max",
                    "action.pitch": "min_max",
                    "action.yaw": "min_max",
                },
            ),
            ]
    
            return ComposedModalityTransform(transforms=transforms)

    2. 新建的DataConfig类注册到ROBOT_TYPE_CONFIG_MAP中。

    图1 DataConfig类注册到ROBOT_TYPE_CONFIG_MAP

    3. 在./starVLA/dataloader/gr00t_lerobot/mixtures.py中注册mixture。

    图2 注册mixture

相关文档