starVLA模型
Notebook创建成功后,需要进入内置算法目录、激活conda环境,并下载所需的模型和数据,为后续的模型二次开发训练做好准备。
进入内置算法目录并激活环境
- 通过JupyterLab或SSH工具接入Notebook实例,打开Terminal终端。
- 进入内置算法目录。
cd /opt/cloud/starvla_ascend_notebook/starVLA
- 激活名为“starvla”的conda虚拟环境。
conda activate starvla
激活成功后,终端命令行前缀会显示“(starvla)”,表示已进入starVLA算法的Python虚拟环境。
下载基模型和数据
通过huggingface-cli工具从huggingface网站下载所需的预训练模型和LeRobot v3.0格式的libero数据,也可以从CloudRobo平台下载v3格式的数据。
- 执行以下命令下载基模型到本地目录。
hf download Qwen/Qwen3-VL-4B-Instruct --local-dir ./playground/Pretrained_models/Qwen3-VL-4B-Instruct hf download StarVLA/Qwen3-VL-4B-Instruct-Action --local-dir ./playground/Pretrained_models/Qwen3-VL-4B-Instruct-Action hf download physical-intelligence/fast --local-dir ./playground/Pretrained_models/fast
- 执行以下命令下载数据到本地目录。
配置数据
以LeRobot v3.0格式的libero数据为例来说明完整的配置流程。
- 在数据的meta目录下添加modality.json。
{ "state": { "x": { "start": 0, "end": 1 }, "y": { "start": 1, "end": 2 }, "z": { "start": 2, "end": 3 }, "roll": { "start": 3, "end": 4 }, "pitch": { "start": 4, "end": 5 }, "yaw": { "start": 5, "end": 6 }, "pad": { "start": 6, "end": 7 }, "gripper": { "start": 7, "end": 8 } }, "action": { "x": { "start": 0, "end": 1 }, "y": { "start": 1, "end": 2 }, "z": { "start": 2, "end": 3 }, "roll": { "start": 3, "end": 4 }, "pitch": { "start": 4, "end": 5 }, "yaw": { "start": 5, "end": 6 }, "gripper": { "start": 6, "end": 7 } }, "video": { "primary_image": { "original_key": "observation.images.image" }, "wrist_image": { "original_key": "observation.images.image2" } }, "annotation": { "human.action.task_description": { "original_key": "task_index" } } } - 若采用datasets.vla_data.data_mix对数据进行混合采样,训练前需要先将数据注册到代码里。
1. 在./starVLA/dataloader/gr00t_lerobot/data_config.py中新建DataConfig类。
class Libero4in1DataConfig: embodiment_tag = EmbodimentTag.FRANKA video_keys = [ "video.primary_image", "video.wrist_image", ] state_keys = [ "state.x", "state.y", "state.z", "state.roll", "state.pitch", "state.yaw", "state.pad", "state.gripper", ] action_keys = [ "action.x", "action.y", "action.z", "action.roll", "action.pitch", "action.yaw", "action.gripper", ] language_keys = ["annotation.human.action.task_description"] observation_indices = [0] action_indices = list(range(8)) state_indices = list(range(-16, 0)) def modality_config(self): video_modality = ModalityConfig( delta_indices=self.observation_indices, modality_keys=self.video_keys, ) state_modality = ModalityConfig( delta_indices=self.state_indices, modality_keys=self.state_keys, ) action_modality = ModalityConfig( delta_indices=self.action_indices, modality_keys=self.action_keys, ) language_modality = ModalityConfig( delta_indices=self.observation_indices, modality_keys=self.language_keys, ) modality_configs = { "video": video_modality, "state": state_modality, "action": action_modality, "language": language_modality, } return modality_configs def transform(self): transforms = [ # action transforms StateActionToTensor(apply_to=self.action_keys), StateActionTransform( apply_to=self.action_keys, normalization_modes={ "action.x": "min_max", "action.y": "min_max", "action.z": "min_max", "action.roll": "min_max", "action.pitch": "min_max", "action.yaw": "min_max", }, ), ] return ComposedModalityTransform(transforms=transforms)2. 新建的DataConfig类注册到ROBOT_TYPE_CONFIG_MAP中。
图1 DataConfig类注册到ROBOT_TYPE_CONFIG_MAP
3. 在./starVLA/dataloader/gr00t_lerobot/mixtures.py中注册mixture。
图2 注册mixture