Training and running AI4EO models requires ingesting large volumes of Earth observation data. The challenge is not only storing those files, but keeping them organized, described, and directly accessible throughout training, validation, and inference.
An AI4EO dataset is a collection of samples. Each sample may contain multiple files organized in folders, while metadata can exist at different levels. Without a shared model for this structure, every dataset requires custom packaging, validation, and loading logic.
TACO packages data and queryable metadata as one unit. Before writing, a producer declares a contract (ฯ, ฮผ) shared by every sample. The writer validates samples against it.
Structure (ฯ) defines the file tree inside
each sample. It supports fixed files such as B02.tif and
bounded sequences such as img*[4,16].tif, useful for time
series and other multitemporal samples with a variable number of files.
Metadata (ฮผ) assigns namespaced, typed fields to sample, folder, and asset levels. Each level becomes a Parquet table that DuckDB can query without an API server.
TACO extensions automatically derive metadata while samples are written. Their results are stored as typed columns and can be queried later.
Structural immutability. The contract (ฯ, ฮผ) is fixed at creation time. Adding new file types or metadata fields requires a new major dataset version. Bounded sequences allow controlled variation, but samples cannot evolve independently. TACO trades open-ended structure for predictable paths, schema validation, and reproducible releases.
CoZIP lets readers locate and fetch individual files without downloading or extracting the complete archive.
Parquet stores metadata in compact columnar tables, so queries read only the rows and columns they need.
Stateless, positional access to files in local storage, cloud providers, or directly inside archives.
FOLDER supports append; ZIP is immutable and optimized for stable releases. Both preserve the same contract and logical dataset structure.
Python
R
Julia
JavaScript
MATLAB
C++
Self-contained. Data and metadata travel together. Contract-first. Structure declared before data is written. Cloud-optimized. Partial reads via byte-range access. Language-agnostic. DuckDB exposes one reader to Python, R, and Julia. Supports FAIR datasets. Description, licensing, providers, and extents are mandatory and explicit.
