Data that keeps its context

01Why a bare number is not enough

Observation data is easy to strip. A value is copied into a spreadsheet, a chart or a model, and on the way it loses the instrument, the clock, the checks and the terms that came with it.

Once stripped, it cannot be trusted or reused safely. Nobody downstream can tell whether a spike was rain or a fault, whether a gap was a calm day or a dead battery, or whether the value may be shared.

The cost of lost context is paid later, and by someone else. An analyst spends days rebuilding what the station knew at the moment of measurement, or uses the value without knowing what it was.

The FAIR principles were written to stop this loss. They ask that data be findable, accessible, interoperable and reusable, by machines as well as by people¹.

For reuse in particular, data and metadata should be richly described, released with a clear usage licence, associated with detailed provenance and meet domain-relevant community standards².

A data product, in this sense, is a value with its context attached and kept attached. The rest of this piece follows that context from the instrument to the user.

02Metadata that people and machines can read

Metadata is how data is found and understood. International geographic standards have long covered metadata so that organisations can describe geospatial data consistently, and it can be discovered, understood, trusted and reused³.

The ISO 19115 family is the core of that work, with parts for an XML implementation of fundamental concepts and for extensions covering acquisition and processing⁴.

Newer profiles follow web practice. The WMO Core Metadata Profile is a profile and extension of OGC API Records, encoded in GeoJSON and made available as crawlable files or through an API⁵.

OGC API Records itself provides a way to browse or search a curated collection of records known as a catalogue, deployed either as static files or behind an API⁶.

W3C DCAT does the same for data catalogues in general⁷. It lets publishers describe datasets and data services with a standard vocabulary, and makes federated search across catalogues in many places possible⁷.

Good metadata is also kept current. When an instrument is moved, replaced or recalibrated, its description changes with it, and earlier records keep pointing to the description that was true at the time.

The practical rule is simple: describe each dataset once, in an open profile, and publish the description where any catalogue can harvest it.

03Provenance and corrections

Provenance is the history of a product: which readings it came from, what was done to them and by whom. Without it, an error found later cannot be traced to the products it affected.

The W3C PROV ontology gives a shared starting point⁸. Its core classes are entity, activity and agent, linked by relations such as generated by, derived from, used and attributed to⁸.

Those few terms are enough to describe most observation chains. A raw reading is an entity, a calibration is an activity, the calibrating laboratory is an agent, and the corrected value is derived from the raw one.

When a calibration turns out to be wrong, provenance shows every product derived from it. Each can then be marked stale, recomputed and published as a new version, with the old version kept and labelled.

Corrections should never erase. A report written last year on the old values remains explainable, because the record shows what was known then and what changed since.

The broader PROV family includes a data model, serialisations and supporting definitions, so provenance can move between systems without being rebuilt⁹.

04Quality flags and honest gaps

Every value should say how far it can be trusted. Open sensor models already make room for this: an observation can carry a statement of result quality and the period in which the result may be used¹⁰.

Quality flags work best from a small controlled vocabulary, applied by named tests. A value marked suspect by a step test tells a user far more than a value silently dropped.

Missing data deserves the same care. A gap should be recorded as a gap, with its reason where known, and never filled with a zero or a copy of the last value.

Uncertainty belongs beside the value wherever the instrument class defines one. Users can then decide whether a reading is good enough for a warning, a design study or a trend.

Flags also travel through processing. A product built from suspect inputs should say so, and a product built with estimated values should name them, so that quality is never upgraded by accident.

Quality checks run in more than one place. A site can catch a stuck sensor at once, while a central check compares neighbours across a region and flags what no single site could see.

05Open formats and APIs

Context only helps if users can reach it with ordinary tools. Open formats and web APIs let the same product serve a forecaster, a planner and a researcher without a custom pipeline for each.

OGC API Features provides building blocks to create, modify and query features on the web, and is defined in parts that each form a separate standard¹¹.

OGC SensorThings provides an open, geospatial-enabled way to manage and retrieve observations and metadata from different sensor systems¹².

For imagery and other files, the STAC specification describes and catalogues spatiotemporal assets, with items as GeoJSON features plus a datetime and links¹³.

For real-time exchange, the WMO Information System uses notification messages over MQTT, so subscribers learn of new data as soon as it is published and fetch it over HTTPS¹⁴.

Versions keep these doors stable. An interface that changes without notice breaks every user at once, so each one carries a version, and old versions are retired only after notice.

Bulk export matters as much as live access. An owner should always be able to take all of its data, with its metadata and provenance, in open formats and leave.

06Residency and licensing travel with the data

Every product needs terms. A clear and accessible usage licence is part of what makes data reusable, and without one a careful user has to leave the data alone².

Standard licence identifiers help machines read those terms. The SPDX License List gives each common licence a short identifier, a full name, the licence text and a permanent address¹⁵.

Some communities set terms for whole classes of data. In the WMO Information System, core data is provided free and unrestricted, without charge and with no conditions on use¹⁴.

Discovery metadata in that system must classify each dataset as core or recommended data under the WMO Unified Data Policy⁵.

When products are combined, the most restrictive upstream term should carry forward, so that a derived map never quietly loosens the conditions of the data inside it.

Residency belongs in the record too. Where data is stored and processed, and under whose jurisdiction, should be stated with each product and enforced by the systems that hold it.

Nexus Observatory Inc. designs data products on these principles, so that each value leaves the site with its context and its terms attached.

Context is not decoration. It is what lets a number from one owner be trusted, combined and reused by another, years after it was measured.