A progressive guide
How Karu Reads
Karu receives exact byte ranges and returns the bytes that belong in each destination buffer. This guide follows that work in order, from the caller description to the final checked completion. The contract stays the same for a local file, an HTTP server, or cloud object storage.
One read, seen from end to end
Imagine a format reader needs 4 KiB from the middle of a large object. It already knows where those bytes begin and where the answer should go. Karu turns that description into one checked completion.
Length 4,096
through 1,052,671
Range and length match
Got 4,096 bytes
Where the bytes live, which range is wanted, and which buffer receives them.
Whether the bytes represent an image, a record, a frame, or something else.
Request preserves the caller intent. The planner turns it into a Transfer, which represents one local read or one HTTP request. Every original request becomes a Part inside that transfer. When work finishes, each part becomes its own Completion.
This separation allows three caller reads to share one network transfer without becoming one caller result. Transport can change while ownership, tags, destinations, and errors remain attached to the original parts.
The boundary of the library
Karu resolves addresses, reports visible size, and reads byte ranges. It supports local files, plain HTTP, Amazon S3 and compatible storage, Google Cloud Storage, Azure Blob and Data Lake Storage, Hugging Face, and Source Cooperative.
It is read only. It does not upload, delete, or list objects. It does not parse file formats and does not cache object data. A layer above Karu decides which bytes have meaning. This narrow contract keeps reads safe to reorder, merge, and retry.
Rumi, for example, maps training windows to compressed frames and submits their byte ranges to Karu. As reads complete, Rumi decodes the frames into arrays or tensors. Karu handles the transport; Rumi handles the format.
A read is a location, not a cursor
A Karu read names an object, an offset, a length, and a destination. The offset travels with the request. There is no shared file position to seek or protect with a lock.
Begin with the nouns
The smallest addressable unit Karu moves. A byte contains eight bits and is represented by one numeric position inside an object.
A finite sequence of bytes with a name. It may be a local file, a web resource, or an object stored by a cloud provider.
The number of bytes from the beginning of the visible object to the first requested byte. Counting starts at zero.
How many consecutive bytes the caller wants. Karu requires a finite, nonzero length for every read.
The memory buffer that receives the result. Its lifetime remains the caller responsibility while the batch is active.
An offset and length considered together. It identifies one contiguous interval without moving or changing the object.
What a cursor is
A traditional file handle often stores a cursor. The cursor is a mutable position inside the file. Reading advances it, and seeking moves it somewhere else. If two threads share that handle, one thread can move the cursor while another is using it, so access must be serialized or each thread needs its own handle.
A positional read does not use that shared position. The offset is an argument of the operation. Reading byte 100 does not alter a value that a later read can observe. This is the meaning of stateless in Karu. Objects describe addresses, while every read carries its own position.
Why a remote object is not an open file
A local file descriptor is a kernel object inside one machine. Opening it produces a reusable handle, and a positional read can copy bytes directly from the filesystem cache or storage device. A remote object has no equivalent open handle. Every read is a network exchange with a service that may require a URL, authentication, and a new response.
This difference changes the dominant cost. A small local read may finish in microseconds. A remote read may wait tens or hundreds of milliseconds even when it returns only a few kilobytes. The delay comes mainly from communication, not from copying the bytes. Karu therefore optimizes request count, concurrency, and connection reuse.
Windows keep coordinates honest
A locator may expose only a window of a larger object. An offset of zero then means the first byte inside that window, not the first byte of the underlying object. Karu converts the request to an absolute offset before transport and rejects any read that leaves the window.
The public read is easiest to reason about as the half open interval [offset, offset + length). HTTP uses an inclusive last byte, so the request builder converts it to bytes=first-last where last = first + length - 1.
Validation happens before the subtraction. Empty reads and an unbounded length are rejected. The planner also checks addition for unsigned 64 bit overflow and verifies that the interval remains inside a bounded locator window.
The same request shape works everywhere
A local path becomes a positional file read. A remote URI becomes a closed HTTP range request. The caller still provides the same four pieces of information and receives the same completion shape.
Resolution creates one canonical identity
A locator parses the caller address into a backend, canonical path, provider container, object key, and optional window. The canonical path is a normalized internal identity. It lets different accepted spellings reach the same configuration rules, planner group, and credential scope.
Resolution does not contact the network and does not load credentials. It only establishes what the resource means. Endpoint selection and authentication happen later, when an actual transfer is prepared.
What an HTTP Range request is
HTTP normally asks a server for an entire resource. A Range request adds a header that names only the needed byte interval. If Karu wants 4,096 bytes beginning at offset 1,048,576, it asks for bytes 1,048,576 through 1,052,671. The final value is inclusive, which is why it is one less than offset plus length.
GET /object HTTP/1.1Host storage.exampleRange bytes=1048576-1052671HTTP/1.1 206 Partial ContentContent-Range bytes 1048576-1052671/2147483648Content-Length 4096Status 206 means the server returned partial content. The Content-Range response header states which bytes arrived and the total object size when known. Karu later checks both values against the transfer it planned. Status 200 means the server returned a normal complete response and may have ignored the range. Status 416 means the requested interval could not be satisfied.
Every operation is independent. That makes concurrency and batching possible without shared cursors.
Give Karu the reads you already know
Remote reads spend much of their life waiting for a round trip. Submitting them one at a time hides the pattern and pays that wait repeatedly. A batch gives the engine enough information to overlap work and find nearby ranges.
Copies request descriptions and returns a batch that can be drained as work finishes.
Returns one completion at a time. Several threads may consume the same batch.
Waits for the whole batch when streaming completions are not useful.
A buffer has one clear lifetime
The destination belongs to one original read even when several reads share a transfer. A caller supplied buffer must remain valid until the batch is released. If the request has no destination, Karu allocates one and returns it through the completion. The caller then releases it with karu_free.
The wait result and the read result are different
One completion is ready. Its own status says whether that individual read succeeded.
The wait ended without a completion. Work continues and the output remains unchanged.
Every original read has completed and the batch is fully drained.
OK means the requested bytes arrived. An error belongs only to that original read.
A batch owns a synchronized completion queue and counts unfinished parts and live transfers. Planner errors enter the same queue as network results, so an invalid range can fail immediately while valid reads from the same submission continue.
Releasing the batch marks pending work as cancelled and waits until active workers no longer reference its buffers. It must not run concurrently with next on that batch.
The batch exposes parallel work without changing the meaning of any individual range.
Fewer requests can move the same useful bytes
Before any input or output begins, the planner validates every read, converts window offsets to object offsets, groups reads by resource, sorts them, and joins neighbours when the trade is sensible.
Four guards keep a merge bounded
The distance between neighbouring ranges must remain small enough.
The combined transfer must stay below the configured size limit.
One transfer may serve only a bounded number of original reads.
Fetched bytes may not grow too far beyond requested bytes.
The defaults allow a 1 MiB gap, a 64 MiB span, 1,024 parts, and at most sixteen fetched bytes for every requested byte. A batch can override these values. Setting the gap to zero disables merging for that batch.
Valid reads are stably sorted by backend, canonical resource, object identity condition, and absolute offset. Only adjacent reads with the same first three values may merge. Different identity conditions remain different HTTP requests even when their byte ranges overlap.
The merge is a linear sweep over each resource group. Every candidate updates the proposed last byte, combined span, useful byte count, and part count. If any guard fails, that transfer closes and the candidate begins the next one.
Arrival reverses the merge
A merged remote response lands in temporary storage. Each part records its position inside that span, its destination, and its tag. Karu copies only the requested segments into the original buffers and reports each read separately. Local reads are never coalesced and go directly to their destinations.
Overlapping reads are valid. Their useful lengths are counted separately for the amplification guard because both destinations still need to be filled. The merged span is stored once, then overlapping bytes are copied into every destination that requested them.
It trades a controlled amount of extra data for fewer round trips without changing any caller visible range.
Local and remote reads share a contract
Once planning is complete, the engine routes each transfer. Local files go to file workers. Remote objects pass through credential resolution and then into the HTTP event loop.
A large batch becomes bounded work
Submitting many reads does not create one thread or one connection for every read. The planner may produce more transfers than the engine can run at once. Extra transfers remain pending, the event loops advance only the configured active limit, and every completion opens capacity for another transfer.
KARU_IO_THREADS defaults to four loops, each with its own thread and connections. They share the client's concurrency limit. More loops can help when TLS, signing or copying saturates one core. With multiple worker processes, compare one and four loops on the full workload, including decoding.
This is backpressure. The amount of work described by the caller can be large while the amount of network work active at one moment remains bounded. The batch completion queue lets the caller consume results at its own pace.
A batch may describe abundant work while queues and the concurrency limit keep active resource use bounded.
A remote read becomes an HTTP exchange
The scheduler decides when a remote transfer may start. The network path then establishes or reuses a secure connection, sends one prepared request, and streams the response back into Karu.
What happens before an HTTPS request
A remote URL begins with a host name that people can recognize. DNS, the Domain Name System, resolves that name to a network address. For direct batch reads on Linux and macOS, Karu caches DNS answers for a minute and rotates addresses across connections. TCP then creates a reliable ordered byte stream between the client and the server. HTTPS adds TLS on top of that stream before HTTP messages are exchanged.
TLS stands for Transport Layer Security. It encrypts traffic so an observer cannot read it, checks that messages were not altered in transit, and validates the server certificate against a trusted certificate authority. The first connection pays for DNS work, a TCP handshake, and a TLS handshake. Reusing the connection avoids repeating most of that setup.
HTTP is a request followed by a response
The request contains a method, a resource path, and headers. Karu uses the GET method because it retrieves data. Headers carry the host, byte range, optional object identity, authentication, and provider specific signing information. A blank line ends the headers. Range reads have no request body.
The response begins with a numeric status, followed by response headers and then the body bytes. Headers arrive before the body, so Karu can decide how the body should be handled while data is still streaming. The engine never treats a successful connection as sufficient. It also verifies the status, response range, and received length.
Remote requests are built late
The locator first resolves to a canonical resource. Karu applies the most specific configuration for that path, obtains credentials when needed, then builds the final URL, headers, byte range, and signature. Building late keeps temporary credentials and region corrections current for each attempt.
The engine sends file transfers to four file workers. Cloud transfers first enter a queue served by four credential workers. Plain HTTP transfers can enter the HTTP queue directly. Resolved cloud work joins that same HTTP queue afterward.
Each event loop thread owns its libcurl multi handle, pending work, retry schedule, active transfer map, and reusable easy handle pool. No other thread mutates those structures. Queue handoff is the boundary between blocking work and event driven network progress.
Each backend produces the same PreparedRequest shape with a URL, headers, closed byte range, HTTP options, and an optional routing region. The transport layer can therefore schedule S3, GCS, Azure, Source, Hugging Face, and plain HTTP without understanding each authentication scheme.
Connections are meant to survive
Reusing a connection avoids DNS lookup and the TCP and TLS handshakes, which can take longer than a small range read.
Each event loop keeps its connections and reusable handles. TLS sessions are shared across loops. Finished handles are reset before reuse so the next transfer starts with its own request options.
An easy handle represents one transfer; a multi handle advances many transfers on one thread. A share handle shares TLS sessions. New connections bypass libcurl's DNS cache and shuffle the resolved addresses to vary endpoint selection.
The multi handle works as an event loop. It asks the operating system which sockets are ready, advances only those transfers, processes completed messages, starts queued work up to the concurrency limit, and sleeps until more progress is possible. This keeps hundreds of waiting requests inexpensive.
HTTP 1.1 and HTTP 2 organize concurrency differently
HTTP 1.1 usually serves one active response per connection, so concurrency often uses several connections. HTTP 2 can multiplex many independent streams over one connection. Multiplexing reduces connection setup, but every stream still shares that connection and its congestion window. The server must also negotiate HTTP 2 during the TLS handshake.
Karu chooses HTTP 1.1 by default and lets configuration request automatic negotiation or HTTP 2. The event loop and multi handle work with either version. Concurrency still describes active transfers, while libcurl decides which connections and streams carry them.
Why Karu sometimes drains unwanted bytes
Stopping halfway through an HTTP body can make the connection unusable because the next response boundary is no longer known. For a small unwanted remainder, Karu reads and discards the bytes so the connection can return to the pool. For a large remainder, it aborts because saving bandwidth is worth paying for a new connection later.
libcurl delivers body fragments through a write callback. The callback returns how many bytes it accepted. Returning zero aborts the transfer, which is useful for a dangerous fallback or oversized error body but normally closes that connection. Karu therefore aborts only when continuing would cost more than reconnecting.
Connections carry prepared transfers. Reuse avoids repeated setup while the event loop advances many waits without creating one thread per request.
A successful response must prove what it contains
A network request completing without an error does not prove that the right bytes arrived. Karu checks the protocol result against the exact range the caller requested.
When a server ignores the range
Some servers answer with the complete object. Karu may discard a small prefix and still deliver the requested interval. It refuses when the discarded prefix would exceed the configured fallback limit, which is 8 MiB by default. This avoids turning a tiny read into an accidental download of a large object.
When a server rewrites the object
GCS can decompress stored gzip objects and ignore Range. Karu sends Accept-Encoding: gzip to retrieve the stored bytes. It rejects responses that report decompression or another transformation, since their byte offsets may have changed.
A partial response must include a parseable Content-Range. Its first byte must equal the transfer offset. Its last byte may equal the requested last byte, or it may be the known final byte of a shorter object. It may never extend beyond the requested interval.
The received body length is checked independently. A correct header followed by a truncated body still fails. A range beginning beyond the object normally produces status 416. The size probe recognizes the special empty object form and reports a size of zero.
Short reads have one precise meaning
If a request reaches the real end of an object, the bytes that exist form a valid prefix. Karu reports that prefix and marks the portion beyond the object with a range error. Other truncation is treated as failure.
How Karu learns an object size
A bounded locator window already knows its visible size. A local file uses filesystem metadata. A remote object uses a one byte GET for byte zero and reads the total size from Content-Range. This deliberately exercises the same authorization and range path as a real read. An empty object answers with the unsatisfied range form and a total of zero.
Remote size results are not cached. The caller decides whether a known size can be reused because only the caller knows how long that answer should remain valid.
Reading both ends of an object
Formats that keep a header at the start and a footer at the end read both with karu_client_read_ends. Karu requests the head and a suffix range for the tail together, so the size, both ends, and the ETag arrive in one round trip. Azure ignores suffix ranges, so there the tail follows the head and carries its ETag as If-Match. Both responses must agree on the size and the ETag.
Several correct reads can still see different versions
Range validation proves that one response contains the requested interval. It does not prove that several responses came from the same version of an object. If another process replaces the object between reads, every response may be individually valid while the caller combines bytes from different versions.
An ETag is a server supplied identifier for one representation of a remote object. When the caller already knows the expected ETag, it may attach that value to every related read. Karu copies the condition at submission and sends it through the HTTP If-Match header.
karu_client_stat returns size and a strong ETag in one probe. Pass that tag as if_match on later reads. Karu also checks the response ETag in case the server ignores the condition; without a strong response tag, enforcement depends on the server.
The probe is optional. Rumi can read frame ranges directly from an external header. That header must belong to the requested dataset version; stat cannot verify the pairing.
The condition is optional and applies only to remote objects. Without it, Karu guarantees correct range delivery for each response but does not create a transactional snapshot across separate requests.
Karu does not equate transport success with data correctness. The response must match the requested byte interval.
Different failures need different next steps
Karu separates ordinary retries from corrections that change the next request. This keeps recovery bounded and prevents permanent failures from looping.
Why a read can be retried safely
A read is idempotent. Repeating it does not modify the object, so one successful attempt has the same intended effect as another. This property permits retries for selected connection failures and temporary service responses. It does not make every failure retryable. Invalid paths, denied access, failed preconditions, and malformed responses return directly to the caller.
Interrupted reads resume with a strong ETag; otherwise they restart the range. Wildcards and tag lists also require a restart because they may accept different versions. If Karu's saved tag stops matching, it discards the partial data. A caller's version mismatch fails the read.
Authentication and signing are related but different
Authentication tells a service which identity is making the request. A bearer token does this by placing a secret token in a header. Request signing instead combines selected request fields with a secret key to produce a signature. The service recalculates that signature and rejects the request if a signed field changed.
AWS Signature Version 4 binds the method, path, query, selected headers, payload description, time, service, and region. That is why Karu must know the final byte range and region before signing. Following a redirect with the old signature would be incorrect and could expose authorization to an unintended host, so signed cloud requests are rebuilt rather than blindly redirected.
A secret or temporary value that proves an identity or permits access.
A cryptographic value bound to one exact request and generated from credential material.
A service location that forms part of an S3 signing scope and request endpoint.
Retries share one deadline
The default allows eight attempts, or three for DNS and connection failures. Credential lookup, requests and retry waits share one deadline. Waits increase with each failure and include random jitter; a server's Retry-After can extend them.
The ordinary attempt counter covers retryable network results and HTTP status such as 408, 425, 429, and server errors. A wrong S3 region has its own one time correction because the region changes the signing key. Expired credentials have another one time correction because the next attempt uses a refreshed identity.
Each path returns the transfer to a different stage. Ordinary retries enter the timed retry list. Region correction returns directly to request preparation. Credential correction invalidates the scoped cache entry and returns to a credential worker. All three remain inside one monotonic deadline.
Connection refusal, name resolution failure, timeout, partial transfer, send or receive failure, and selected TLS or HTTP 2 failures are retryable transport results. HTTP status 408, 425, 429, and server errors are also retryable. The retry decision uses structured result codes rather than wording from platform dependent error messages.
The outer request timeout defaults to 120 seconds. A separate 30 second connect timeout bounds connection establishment. Low speed protection aborts a connection that remains open but transfers fewer than 1,024 bytes per second for 60 seconds. Before sleeping, Karu verifies that enough outer deadline remains for another attempt.
Failure bodies are bounded
Karu retains only the first 4 KiB of an error body for classification and diagnostics. It can drain up to 256 KiB of unwanted response data to preserve a reusable connection, but aborts beyond that point. Human readable messages collapse whitespace, limit the body summary, and redact query strings before reaching logs.
Retries are bounded by attempts and time. Permanent failures return a stable status and a redacted message.
Credentials are reused only where they are valid
Cloud reads may need temporary proof of identity. Karu obtains that proof outside the network event loop, scopes its reuse to matching resources, and refreshes it before it expires.
Different providers discover different credentials
Credentials can come from explicit options, profiles, files, metadata services, or an application callback. Karu caches a usable result by its effective path scope and refreshes it near expiry. Concurrent requests for the same cold scope share one lookup instead of starting many identical ones.
Explicit keys come first, followed by profiles, credential processes, web identity, container credentials, and instance metadata when discovery is enabled.
Static tokens or interoperable keys come before Application Default Credentials and the compute metadata service.
A connection string, shared key, SAS, or token can authenticate directly. Service principals, workload identity, and managed identity provide renewable access.
Hugging Face uses an environment token or CLI token file. Source is anonymous for public data and can use its own profile or callback for signed access.
Native discovery keys the cache by provider family and by the configuration origins that supplied credential options for the canonical path. Two paths share a credential only when those origins resolve to the same scope.
A custom callback may return a canonical path prefix that states where its value is valid. Karu checks that the requested path is inside that prefix before caching it. Without a prefix, the callback runs for every request, which is useful when an external SDK already owns its cache.
Refresh is lazy and proactive. A temporary credential becomes refreshable during the final tenth of its lifetime, with that margin bounded between one and sixty seconds. The first request inside the window performs the refresh while matching requests wait for the same result. A failed lookup is returned but never stored as a negative cache entry.
An application callback can provide credentials from a vendor SDK or secret manager. It receives the canonical path and may return an expiry and a canonical cache prefix. Calls for different paths may run concurrently, so the callback must be thread safe and must place its own bound on external input and output.
A reusable credential must match the requested path and remain valid long enough for the next attempt.
Keep the engine alive and tune from evidence
Most remote range workloads are limited by latency rather than local compute. Reuse one client for related work, then tune the amount of active work, the planner trade, and the recovery budget from measurements.
Use one client for related work
A client owns the engine, handle pool, connection reuse, and credential cache. Creating one client per read throws away the state that makes remote reads efficient. Share a client across threads and submit meaningful batches.
The client captures an immutable configuration snapshot when it is created. Reads can share the client without observing a builder that changes underneath them. Path rules are resolved against each canonical path, while network and planner defaults stay stable for the lifetime of that client.
This is why changing a configuration builder does not mutate an existing client. Create a new client when transport policy or provider configuration must change.
Tune only after reuse is working
Controls how many remote transfers may be active. The default is 64.
Raise it when latency leaves the connection idle. Lower it when the service or process needs a tighter bound.Controls the trade between request count and extra bytes.
Measure fetched bytes as well as requested bytes. High useful throughput can hide wasteful merging.Bound the lifetime of one logical read and the number of tries inside it.
A retry is useful only when enough deadline remains for it to finish.Path rules make one client useful for many resources
Configuration lookup begins with the longest matching canonical path. A bucket rule therefore wins over a global option for objects inside that bucket, and a deeper prefix can override the bucket rule for one subtree. If no path rule supplies a value, Karu checks the global option, then the captured environment value, then the built in default.
The same canonical path also limits credential reuse. This alignment prevents an option selected for one resource from silently authorizing another resource with a different scope.
Forked processes rebuild safely
A child process does not inherit the parent engine threads in a usable form. Karu detects the process change on first use and creates a fresh engine. The child gets its own connections and workers without touching stale thread state.
Submit batches in the process that will consume them. Inherited batches return KARU_ERR_INVALID in the child, and freeing one leaves it allocated until process exit.
The C ABI and the C++ facade
The stable public boundary is a C application binary interface. Opaque pointers hide the C++ implementation, numeric status values cross the boundary without exceptions, and callers release objects through matching Karu functions. This shape is suitable for language bindings because it does not expose the C++ standard library in the binary contract.
The C++ facade adds move only ownership, spans, optional planner overrides, expected results, and automatic cleanup. It does not create another engine. Both interfaces reach the same client, locator, batch, and completion machinery.
What Karu deliberately does not cache
Karu reuses connections, TLS sessions, and credentials because they support many reads. It does not cache object bytes or remote size results. A data cache would need eviction, invalidation, memory budgets, and a workload specific policy. Keeping it outside Karu lets the format reader or application choose whether caching is useful.
Measure the two sides of performance
Planner benchmarks should report transfer count, fetched bytes, requested bytes, and planning time. Network benchmarks should separate cold connection setup from steady state and use a real remote endpoint. A local server removes the latency that batching and concurrency are designed to hide.
Concepts at a glance
- Backend
- The provider specific component that converts one resolved object and range into a prepared request.
- Backpressure
- A bound that lets a large batch wait in queues instead of becoming unlimited active network work.
- Completion
- The result of one original read with its tag, status, received byte count, and destination.
- Canonical path
- The normalized object identity used for configuration matching, grouping, and credential scope.
- Concurrency
- The number of remote transfers allowed to make progress at the same time.
- ETag
- A remote object version identifier that can make several reads fail if the object no longer matches.
- Event loop
- One loop that advances many nonblocking network operations when their sockets become ready.
- Half open range
- An interval that includes its first byte and excludes its end, written as offset through offset plus length.
- Idempotent
- Safe to repeat without changing stored data. Karu reads have this property.
- Latency
- The waiting time between sending a request and receiving its response.
- Round trip
- One journey from client to server and back. Network setup and requests require one or more of them.
- Single flight
- One shared lookup used to satisfy multiple simultaneous requests for the same credential scope.
- Stateless read
- A read whose position is carried in the operation rather than stored in a shared cursor.
- Transfer
- One local read or HTTP request that may serve several original caller reads.