How Rumi Reads

One read, seen from end to end

Suppose a model asks for one spatial window, one time position, and two bands. The public call is small because the implementation carries those choices through every layer.

chip = rumi.read(
    "scene.rumi",
    header,
    time=[2],
    bands=[1, 3],
    window=(512, 256, 256, 256),
    framework="torch",
)

Turn a user request into native values

rumi.read() is synchronous: it returns a finished array or raises an exception. Its arguments describe three different things—the bytes to read, the samples to keep, and the array to return.

The data file and header are separate on purpose. Rumi can plan exact byte ranges from the small header before touching the potentially large data source. A caller that already has the header can therefore begin a remote read without first downloading metadata from the data object.

read() in bindings/python/rumi/_read.py parses the external header, validates the selection and resolves the framework consumer before opening the source. The native handle wrappers live in bindings/python/rumi/_native.py; the CFFI declarations and library loading remain isolated in _ffi.py.

spec = _Spec(header)
consumer = resolve_framework(framework, spec.fields.dtype)
times = _resolve_axis(time, "time", spec.fields.time_count)
bands = _resolve_axis(bands, "bands", spec.fields.samples_per_pixel)
window = _resolve_window(window, spec.fields.image_length, spec.fields.image_width)
pattern = _resolve_pattern(pattern)
arr = _read_one(_Source(source), spec, times, bands, window, pattern)
return consumer.convert(arr)

_resolve_axis() expands a half-open range when necessary, validates Python’s zero-based band and time indices, and converts them to the C API’s one-based convention. A null pointer plus count zero means “all.” _resolve_window() turns (row, column, height, width) into the row and column offsets and sizes expected by C.

_Source owns the native source handle. For in-memory input, the C++ MemorySource only borrows the Python buffer, so _Source._keep keeps that buffer alive until the native source is released. _Spec calls rumi_spec_parse(), arranges for the native object to be freed with ffi.gc(), and exposes the parsed fields needed for Python-side bounds checks.

Share a stable C boundary, then allocate exactly once

CFFI, Python’s C foreign-function bridge, calls the public C API in core/src/capi.cpp. That API deliberately exposes plain handles, integers, pointers, and status codes—not C++ classes or exceptions.

prepare_selection() centralizes pointer/count checks, selected-axis resolution, output-layout compilation, and overflow-safe size calculation. Both rumi_read(), where the caller supplies storage, and rumi_read_dlpack(), where Rumi allocates storage, use this same preparation. The two entry points cannot silently disagree about shape or required bytes.

The read path allocates exactly need bytes, invokes read_window(), and only then wraps the allocation with finish_dlpack(). Every failure path frees the allocation. The dtype registry supplies the decoded byte width and exact DLPack code independently from the logical width stored in the file.

Compile the selection into frame tasks

With shape and storage settled, the core turns the logical selection into one FrameTask for every compressed frame that can contribute samples.

The header’s frame_unit records a choice made when the file was written: a frame may hold several bands or times, or those axes may select separate frames. append_read_plan() walks an axis only when that axis changes the frame index. Selected planes that live together remain members of one task, avoiding a separate fetch for every plane inside that frame.

The function walks only the tile rows and columns intersecting the window. For each relevant band and time combination it calls Header::frame_index(), then obtains the compressed range with frame_offset() and frame_byte_count(). It also precomputes where every selected plane belongs in the output.

idx = h.frame_index(tile_row, tile_col, band, time)
task.offset = h.frame_offset(idx)
task.compressed_size = h.frame_byte_count(idx)

When a complete decoded frame maps byte-for-byte to one contiguous output region, the task stores a direct destination. Otherwise it stores crop coordinates and strides for a later copy_rect(). No transport decision has been made yet.

Ask for byte ranges, not raster concepts

Karu is a read-only transport library. Rumi gives it a locator plus offset and length; Karu returns exactly that byte range. It does not know about bands, tiles, frames, or output arrays.

A locator is Karu’s transport-independent description of where bytes live. karu_resolve() turns a local path, HTTP URL, or supported cloud URI into that description once when TransportSource::open() runs. Reads are positional rather than cursor-based, so independent frame tasks can request different parts of the same object concurrently.

execute_plan() asks each source for remote_locator(). A memory source returns no locator and later performs a bounded memcpy. A local TransportSource also joins local_tasks; its Rumi worker calls the blocking convenience function karu_client_fetch(). Only a genuinely remote locator enters the asynchronous batch route.

For remote work, Rumi creates one karu_req per frame and stores &task in its caller-controlled tag. It submits the complete remote workload before decoding local tasks. Seeing every range at once lets Karu schedule and coalesce compatible requests while Rumi’s CPU workers remain busy.

karu_batch_next() returns completions as they become ready. Rumi checks the Karu status and exact byte count, recovers the FrameTask from the tag, attaches the compressed buffer, and sends the ready tasks to the decoder in waves. Karu’s I/O threads continue filling the next wave while Rumi decodes the current one.

The API is synchronous; the engine is concurrent

There is no await rumi.read(...). The Python call blocks until every requested sample is ready. The asynchronous work lives underneath that call, where Karu can progress remote I/O while Rumi workers read local frames and decode completed ones.

Which thread owns what?

karu_batch_next(batch, -1) blocks the coordinator until the first completion is available. Rumi then calls karu_batch_next(batch, 0) repeatedly to drain everything already ready without waiting. That group becomes one decoder wave. The loop repeats until Karu reports the batch ended.

The Rumi thread pool is process-wide and is created only when a plan has more than one frame task. Its default size is one unless RUMI_NUM_THREADS or rumi.set_num_threads() changes it; without a pool, Executor::run() performs the same tasks serially on the calling thread. The first genuinely parallel read fixes the pool size, so configuration must happen before that read. Each call submits its own ThreadPool::Batch and waits only for that batch.

Use one OpenZL runtime extended by GeoZL

OpenZL is the compression runtime that understands the graph stored inside a compressed frame. GeoZL is an extension library that adds raster-oriented transform nodes to that runtime. They are one decoding path, not two competing decoders.

An OpenZL frame carries the graph and parameters needed to reconstruct its values. Here, a graph is a recipe of connected transform nodes: the output of one operation becomes the input of another. Built-in nodes implement general operations; GeoZL contributes nodes designed for geospatial raster data. Rumi does not choose or rebuild that graph during a read—it supplies the compressed frame to OpenZL and validates the result against the Rumi header.

struct WorkerState {
    ZL_DCtx* dctx = ZL_DCtx_create();

    WorkerState() {
        geozl_register_decoders(dctx);
    }
};

WorkerState in core/src/plan.cpp is declared thread_local: every worker thread receives its own instance the first time it decodes. The worker can reuse its OpenZL context plus compressed and decoded scratch buffers for later tasks, but no two threads share that mutable decoder state. Frames remain independent; reuse avoids rebuilding the runtime and reallocating scratch for every frame.

execute_task() calls ZL_DCtx_decompressTyped(), then checks that OpenZL produced numeric samples with the expected fixed width and exact decoded byte size. Boolean frames receive one additional check: every decoded byte must be zero or one. Only a validated frame can reach output placement.

Decode directly or copy the requested rectangle

The output buffer and all of its strides were fixed before the plan ran. A completed frame therefore already knows whether it can decode directly into its final location or needs a temporary decoded frame.

The output pattern determines shape and strides. For example, "t y x b" places time first and band last; another order changes where a task must write without changing which source frames it needs. This is why compile_layout() runs before append_read_plan().

rumi_read_dlpack() allocates the exact output size before calling the core. finish_dlpack() wraps that allocation in a versioned DLPack tensor. DLPack is a small ownership-and-metadata protocol shared by array libraries: it describes the address, shape, strides, dtype, and device without copying the decoded array into a second framework-owned buffer.

Python wraps the native tensor in RumiArray. Calling numpy(), torch(), jax(), or tensorflow() transfers that allocation through DLPack. RumiArray.__dlpack__() permits the transfer exactly once; if no consumer takes it, the wrapper frees the native tensor itself. Framework compatibility is checked before any source is opened.

Read several images in one plan

read_many() does not loop over read(). It pairs each source with its header and, when supplied, its window, then sends the whole batch through one read_items() call and one execution plan. Without windows, it reads every image in full.

batch = rumi.read_many(
    ["north.rumi", "south.rumi"],
    [north_header, south_header],
    windows=[(0, 0, 256, 256), (512, 0, 256, 256)],
    bands=[1, 3],
    framework="torch",
)

compatible_headers() requires the same tile size, band count, dtype, time count, and frame-axis behavior. A batch without windows also requires equal image dimensions. With explicit windows, the source images may differ in size, but every window must have the same height and width so the results fit one rectangular tensor.

The combined plan is the performance feature. Remote frame requests from every item are visible to one Karu submission, local and memory work share the same Rumi pool, and every ready frame can decode without waiting for earlier items to finish.

The read path by file

These are the implementation files to open in order when changing or debugging a read.