Help Center/ PanguLargeModels/ User Guide/ Using Data Engineering to Create a Dataset/ Dataset Format Requirements/ Format Requirements for Weather Datasets
Updated on 2025-07-02 GMT+08:00
View PDF
Share

Format Requirements for Weather Datasets

ModelArts Studio supports the import of weather datasets, including marine meteorological data.

Marine meteorological data is usually derived from weather reanalysis. Weather reanalysis uses modern weather models and data assimilation technologies to reprocess historical observation data and generate high-quality weather records. The data can be global or region-specific, aiming to provide complete, consistent, and high-resolution weather data.

The reanalysis data is in binary format. For details about the format requirements, see Table 1.

Table 1 Format requirements for weather datasets

File Content

File Format

Example

Meteorology - Weather data

nc, cdf, netcdf, gr, gr1, grb, grib, grb1, grib1, gr2, grb2, and grib2
  • Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited.

Weather data usually includes global or regional weather variables, such as temperature (T), atmospheric pressure (P), and wind speed (U and V). In a file, these variables may be arranged by time, geographical range, and pressure level. The following is an example:

{"geo_range": {"lat": ["-90.0", "90.0"], "lon": ["0.0", "360.0"]}, "time_range": ["1640995200000", "1641164400000"], "total_size": 7376211808, "surface_features": ["P", "T", "U", "V"], "upper_air_layers": ["1000hPa", "100hPa", "150hPa", "175hPa", "200hPa", "250hPa", "300hPa", "400hPa", "500hPa", "50hPa", "600hPa", "700hPa", "850hPa", "925hPa"], "upper_air_features": ["Q", "T", "U", "V", "Z"]}
  • geo_range: defines the geographical range covered by data. The value of lat ranges from -90.0 to 90.0, and the value of lon ranges from 0.0 to 360.0.
  • time_range: time range of data. The timestamp is in milliseconds.
  • total_size: total size of the files to download, in bytes.
  • surface_features: list of surface variables, such as atmospheric pressure (P), temperature (T), and wind speed (U and V).
  • upper_air_layers: upper-air pressure levels, for example, 1000 hPa and 100 hPa.
  • upper_air_features: upper-air variables, for example, humidity (Q), temperature (T), wind speed (U and V), and height (Z).

Meteorology - Ocean data

nc, cdf, netcdf, gr, gr1, grb, grib, grb1, grib1, gr2, grb2, and grib2
  • Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited.
  • Ocean data usually contains global or regional ocean variables, such as temperature (T), atmospheric pressure (P), and wind speed (U and V). The following is an example of the format: 
    {"geo_range": {"lat": ["-90.0", "90.0"], "lon": ["0.0", "360.0"]}, "time_range": ["1640995200000", "1641164400000"], "total_size": 7376211808, "surface_features": ["SSH", "T", "P", "U", "V"], "under_sea_layers": ["0m", "6m", "10m", "20m", "30m", "50m", "70m", "100m", "125m", "150m", "200m", "250m", "300m", "400m", "500m"], "under_sea_features": [ "T", "U", "V", "S"]}
    • geo_range: defines the geographical range covered by data. The value of lat ranges from -90.0 to 90.0, and the value of lon ranges from 0.0 to 360.0.
    • time_range: time range of data. The timestamp is in milliseconds.
    • total_size: total size of the files to download, in bytes.
    • surface_features: list of sea surface variables, such as sea surface height (SSH), temperature (T), and wind speed (U and V).
    • under_sea_layers: deep-sea pressure levels, for example, 500 m and 400 mPa.
    • under_sea_features: list of deep-sea variables, such as sea salt (S), temperature (T), and sea flow rate (U and V).

Meteorology - Ecological data

nc, cdf, netcdf, gr, gr1, grb, grib, grb1, grib1, gr2, grb2, and grib2
  • Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited.

Ecological data usually includes ecological variables such as total chlorophyll concentration (Tca), chlorophyll concentration (Chl), and diatom concentration (Dia). The following is an example:

{"geo_range": {"lat": ["-90.0", "90.0"], "lon": ["0.0", "360.0"]}, "time_range": ["1640995200000", "1641164400000"], "total_size": 7376211808, "surface_features": ["Tca","Chl","Dia","Coc","Cya","Irn","Nit","MLD"]}
  • geo_range: defines the geographical range covered by data. The value of lat ranges from -90.0 to 90.0, and the value of lon ranges from 0.0 to 360.0.
  • time_range: time range of data. The timestamp is in milliseconds.
  • total_size: total size of the files to download, in bytes.
  • surface_features: ecological variables, for example, Tca, Chl, and Dia.

Meteorology - Wave data

nc, cdf, netcdf, gr, gr1, grb, grib, grb1, grib1, gr2, grb2, and grib2
  • Import from OBS: The size of a single file cannot exceed 50 GB, and the number of files is not limited.

Wave data usually includes the significant wave height (SWH). The following is an example:

{"geo_range": {"lat": ["-90.0", "90.0"], "lon": ["0.0", "360.0"]}, "time_range": ["1640995200000", "1641164400000"], "total_size": 7376211808, "surface_features": ["SWH"]}
  • geo_range: defines the geographical range covered by data. The value of lat ranges from -90.0 to 90.0, and the value of lon ranges from 0.0 to 360.0.
  • time_range: time range of data. The timestamp is in milliseconds.
  • total_size: total size of the files to download, in bytes.
  • surface_features: wave variable, which is significant wave height (SWH).