Datasets

This page provides an overview of how datasets are defined, structured, and handled in the processing chain. The goal is to ensure consistent, analysis-ready data products that can be accessed efficiently and extended over time.

What is a Dataset?

A dataset is the main unit of storage and analysis. A dataset:

• represents a rectangular collection of variables with shared coordinates

• is stored in Zarr format

• corresponds to one deployment of an instrument or product

Datasets are designed so that they can be accessed directly for analysis via our intake catalog.

Analysis-Ready Datasets

Our goal is to produce analysis-ready datasets. These datasets should be usable for scientific analysis without additional preprocessing.

An analysis-ready dataset should:

• represent the whole observation period

• follow CF conventions where possible

• include appropriate metadata and units

Dataset Organization

Datasets follow a hierarchical naming scheme that reflects their observational context.

The hierarchy is:

platform.campaign.instrument

or, if no campaign is relevant:

platform.instrument

The instrument name should include configuration (_c1) and version (_v1) information.

Examples:

BCO.surfacemet_wxt_v1
BCO.lidar_CORAL_LR_t_c1_v1
METEOR.EUREC4A.lidar_LICHT_LR_t_v1

Naming rules:

• . separates hierarchical levels

• _ is used within names

Datasets can be accessed through the intake catalog, for example:

import intake


cat = intake.open_catalog("https://tcodata.mpimet.mpg.de/catalog.yaml")
cat.BCO.surfacemet_wxt_v1.to_dask()

/builds/tco/bco/docs/.venv/lib/python3.12/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 8GB
Dimensions:      (time: 43702710, bnd: 2)
Coordinates:
    alt          float64 8B ...
    lat          float64 8B ...
    lon          float64 8B ...
  * time         (time) datetime64[ns] 350MB 2010-12-16T16:24:00 ... 2026-04-...
Dimensions without coordinates: bnd
Data variables: (12/22)
    DIR          (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    DL           (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    DR           (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    H            (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    HDS          (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    HI           (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    ...           ...
    TI           (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    VEL          (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    VH           (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    VR           (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    VS           (time) float64 350MB dask.array<chunksize=(262144,), meta=np.ndarray>
    time_bounds  (time, bnd) datetime64[ns] 699MB dask.array<chunksize=(262144, 2), meta=np.ndarray>
Attributes:
    Conventions:           CF-1.12
    _logical_cutoff_date:  2026-04-02T00:00:00Z
    bcoproc_version:       0.0.0.post1246.dev0+39062d5
    featureType:           timeSeries
    institution:           Max Planck Institute for Meteorology, Hamburg
    license:               CC0-1.0
    location:              The Barbados Cloud Observatory (BCO), Deebles Poin...
    platform:              BCO
    source:                Vaisala WXT-520
    summary:               This dataset contains basic meteorological measure...
    title:                 WXT-2 ground station data from BCO (Level 1)
    tool_versions:         {"Python": "3.11.2 (main, Apr 28 2025, 14:11:48) [...

xarray.Dataset

• Dimensions:
  • time: 43702710
  • bnd: 2
• Coordinates: (4)
  • alt
    ()
    float64
    ...

    long_name :

    height above mean sea level

    standard_name :

    altitude

    units :

    m

    [1 values with dtype=float64]

  • lat
    ()
    float64
    ...

    long_name :

    north latitude

    units :

    degree_north

    valid_max :

    90.0

    valid_min :

    -90.0

    [1 values with dtype=float64]

  • lon
    ()
    float64
    ...

    long_name :

    east longitude

    units :

    degree_east

    valid_max :

    180.0

    valid_min :

    -180.0

    [1 values with dtype=float64]

  • time
    (time)
    datetime64[ns]
    2010-12-16T16:24:00 ... 2026-04-...

    axis :

    T

    bounds :

    time_bounds

    standard_name :

    time

    units_metadata :

    leap_seconds: utc

    array(['2010-12-16T16:24:00.000000000', '2010-12-16T16:24:10.000000000',
           '2010-12-16T16:24:20.000000000', ..., '2026-04-01T23:59:31.000000000',
           '2026-04-01T23:59:41.000000000', '2026-04-01T23:59:50.000000000'],
          shape=(43702710,), dtype='datetime64[ns]')

• Data variables: (22)
  • DIR
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: circmean

    long_name :

    wind direction

    standard_name :

    wind_from_direction

    units :

    degree

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • DL
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: circleft(DIR)

    long_name :

    extreme left wind direction

    units :

    degree

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • DR
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: circright(DIR)

    long_name :

    extreme right wind direction

    units :

    degree

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • H
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: sum

    long_name :

    sum of hail within 10 sec time bin

    units :

    cm-2

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • HDS
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: sum

    long_name :

    duration of hail in second

    units :

    s

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • HI
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    mean hail intensity

    units :

    cm-2 h-1

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • MNV
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: minimum

    long_name :

    minimum wind speed

    units :

    m s-1

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • MXV
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: maximum

    long_name :

    maximum wind speed

    units :

    m s-1

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • P
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    pressure

    standard_name :

    air_pressure

    units :

    hPa

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • R
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: sum

    long_name :

    sum of rain within 10 sec time bin

    standard_name :

    rainfall_amount

    units :

    mm

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • RDS
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: sum

    long_name :

    duration of rain

    units :

    s

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • RH
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    relative humidity in percent

    standard_name :

    relative_humidity

    units :

    0.01

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • RI
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    rain intensity, running mean over one minute before timestamp

    units :

    mm h-1

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • SDQ
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    data quality in %

    units :

    0.01

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • T
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    temperature

    standard_name :

    air_temperature

    units :

    degree_Celsius

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • TH
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    heater temperature

    units :

    degree_Celsius

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • TI
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    internal temperature

    units :

    degree_Celsius

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • VEL
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    wind speed

    standard_name :

    wind_speed

    units :

    m s-1

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • VH
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    heater voltage

    units :

    V

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • VR
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    comment :

    nominally 3.5V

    long_name :

    reference voltage

    units :

    V

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • VS
    (time)
    float64
    dask.array<chunksize=(262144,), meta=np.ndarray>

    cell_methods :

    time: mean

    long_name :

    supply voltage

    units :

    V

    Array Chunk Bytes 333.43 MiB 2.00 MiB Shape (43702710,) (262144,) Dask graph 167 chunks in 2 graph layers Data type float64 numpy.ndarray

  • time_bounds
    (time, bnd)
    datetime64[ns]
    dask.array<chunksize=(262144, 2), meta=np.ndarray>

    Array Chunk Bytes 666.85 MiB 4.00 MiB Shape (43702710, 2) (262144, 2) Dask graph 167 chunks in 2 graph layers Data type datetime64[ns] numpy.ndarray

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2010-12-16 16:24:00', '2010-12-16 16:24:10',
                   '2010-12-16 16:24:20', '2010-12-16 16:24:30',
                   '2010-12-16 16:27:50', '2010-12-16 16:28:00',
                   '2010-12-16 16:28:10', '2010-12-16 16:28:20',
                   '2010-12-16 16:28:30', '2010-12-16 16:30:20',
                   ...
                   '2026-04-01 23:58:20', '2026-04-01 23:58:30',
                   '2026-04-01 23:58:40', '2026-04-01 23:58:50',
                   '2026-04-01 23:59:00', '2026-04-01 23:59:10',
                   '2026-04-01 23:59:20', '2026-04-01 23:59:31',
                   '2026-04-01 23:59:41', '2026-04-01 23:59:50'],
                  dtype='datetime64[ns]', name='time', length=43702710, freq=None))

• Attributes: (12)

  Conventions :

  CF-1.12

  _logical_cutoff_date :

  2026-04-02T00:00:00Z

  bcoproc_version :

  0.0.0.post1246.dev0+39062d5

  featureType :

  timeSeries

  institution :

  Max Planck Institute for Meteorology, Hamburg

  license :

  CC0-1.0

  location :

  The Barbados Cloud Observatory (BCO), Deebles Point, Barbados, West Indies

  platform :

  BCO

  source :

  Vaisala WXT-520

  summary :

  This dataset contains basic meteorological measurements from a multi-sensor weather instrument is permanently installed on top of the office container at the BCO.

  title :

  WXT-2 ground station data from BCO (Level 1)

  tool_versions :

  {"Python": "3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0]", "click": "8.1.8", "frozenlist": "1.6.0", "idna": "3.10", "locket": "1.0.0", "attrs": "25.3.0", "six": "1.17.0", "aiohttp": "3.11.18", "aiosignal": "1.3.2", "packaging": "25.0", "cftime": "1.6.4.post1", "numpy": "2.2.5", "ruff": "0.13.0", "pytz": "2025.2", "dask": "2025.4.1", "python-dateutil": "2.9.0.post0", "partd": "1.4.2", "bcoproc": "0.0.0.post1246.dev0+39062d5", "toolz": "1.0.0", "pandas": "2.2.3", "pyyaml": "6.0.2", "tzdata": "2025.2", "netcdf4": "1.7.2", "xarray": "2025.4.0", "propcache": "0.3.1", "iniconfig": "2.1.0", "certifi": "2025.4.26", "multidict": "6.4.3", "aiohappyeyeballs": "2.6.1", "yarl": "1.20.0", "datarecords": "0.0.0.post1246.dev0+39062d5", "zipp": "3.21.0", "pytest": "8.3.5", "cloudpickle": "3.1.1", "bcologs": "0.0.0.post1246.dev0+39062d5", "pluggy": "1.5.0", "pendulum": "3.1.0", "fsspec": "2025.3.2", "importlib_metadata": "8.7.0"}

Coordinate Conventions

Datasets follow CF conventions for coordinate naming where possible.

Primary coordinates include:

Coordinate	Meaning
time	UTC timestamp
alt	altitude above the geoid (meters)
range	line-of-sight distance from instrument (meters)
lat	latitude (degrees_north)
lon	longitude (degrees_east)

Primary coordinates must:

• be strictly monotonic

• contain no missing values

When sensor and data coordinates differ, sensor coordinates are provided using the prefix sensor_, e.g., sensor_alt.

Incrementally Growing Datasets

Datasets are stored as Zarr archives and are extended continuously as new data becomes available. Rather than rewriting datasets, new data is appended as additional chunks. This enables efficient cloud-based storage and scalable analysis. The processing chaing orchestrated by an Airflow server.