OpenLake is an Apache-2.0 distributed storage engine written in **Rust on `io_uring`**, built for one job: keeping GPUs fed during LLM inference and training. The repository opened on **27 April 2026** and has since reached roughly **2,309 stars and 420 forks**, shipping steadily through **v0.6.0** (21 July 2026) with v0.7 work already landing in main. The headline claim on the README is **million+ IOPS within 1 ms** — cache-like performance while remaining fully persistent and durable. ## Why a Storage Engine Is an Inference Problem The expensive part of serving long-context LLMs is prefill. Every time a request arrives with a 128K-token prompt, the model recomputes the KV cache for tokens it may have already processed for a previous request. GPU memory is far too small to hold that cache at any meaningful scale, and conventional object storage is far too slow to serve it back inside a latency budget — so the cache gets thrown away and recomputed. OpenLake's core proposition is a **petabyte-scale KV cache pool co-located on the GPU hosts themselves**. Each GPU node contributes its host RAM and NVMe to a shared cluster. The inference engine writes the KV once and reads it back in milliseconds, and a prefix computed on one GPU host can be served to any other node from the shared pool. The project reports a **66× speedup on time to first token** for cached 128K-context requests. ## Dropping It Into vLLM The integration path is deliberately minimal — no code changes, just a connector and a config flag: ```bash pip install openlake-vllm openlaked export PYTHONHASHSEED=0 vllm serve <model_name> --kv-transfer-config '{"kv_connector":"OpenLakeConnector", ...}' ``` By default `openlaked` offloads to the same host. Scaling to a fleet means starting each node with an RDMA config (`kv_rdma_0.toml`, `kv_rdma_1.toml`, …) and pointing vLLM workers at the node list in ID order. The README's demonstration runs Gemma4-31B on an H100 at a 256K context window with OpenLake enabled versus disabled. ## Five Workloads, One Engine KV cache offload is the flagship, but the same engine is pitched at the adjacent bottlenecks: | Workload | What OpenLake provides | |---|---| | **KV cache offload** | Petabyte-scale cache co-located on GPU hosts, cutting prefill cost | | **VectorDB** | Fast index building and vector serving | | **Checkpointing** | Ultra-fast checkpoint write and restore for RL and ML runs | | **Model training** | Small-file I/O and fast random reads to cut GPU idle time | | **Context storage** | Massive conversation and memory stores for agentic retrieval | It also runs as a plain **S3-compatible object store**. After a `cargo build --release`, any AWS CLI or S3 client can talk to it on port 9000 — `aws --endpoint-url http://127.0.0.1:9000 s3 cp ./checkpoint.safetensors s3://demo/` works with no adaptation. ## Architecture The design choices are all aimed at shortening and stabilizing the path from storage to GPU memory: - **Zero copy.** With GPUDirect Storage and RDMA, data moves directly between NVMe or an RDMA NIC and GPU memory, skipping host-memory staging and the page cache entirely. - **Core-local asynchronous I/O.** One pinned [`compio`](https://github.com/compio-rs/compio) runtime per physical core, backed by Linux `io_uring`. Requests stay on the same core through the hot path, avoiding work stealing and cross-core contention — the discipline that makes the sub-millisecond tail latency claim plausible rather than a best-case average. - **Burst-aware RDMA.** PacedRDMA applies credit-based flow control so senders cannot overwhelm receivers, protecting tail latency during request bursts. - **Efficient durability.** SIMD Reed-Solomon erasure coding spreads data and parity across drives, giving durability at lower capacity overhead than full replication. That last point is what separates OpenLake from a cache: losing a node does not lose the data. ## Caveats OpenLake is **Linux-first by design**, and the reason is architectural rather than incidental — `io_uring`, GPUDirect Storage, and RDMA are Linux interfaces. macOS is supported for development only and explicitly cannot do RDMA; Windows goes through WSL2. Anyone evaluating on a laptop is testing the TCP path, not the one the benchmarks describe. The headline performance also assumes hardware most teams do not have on hand. The full picture needs NVMe with GPUDirect Storage support and an InfiniBand or RoCE fabric; on commodity TCP networking the numbers will look different, and the README does not publish that comparison. The KV-cache quickstart is a clean `pip install`, but the object store still requires building from source with a system dependency list (`clang`, `cmake`, `libhwloc-dev`, `libudev-dev`) and a Rust 1.91+ toolchain — there is no prebuilt binary or container image in the quickstart path. Maturity is the other consideration: this is a **v0.x project three months old** with 112 open issues, and recent commits include fixes to fundamentals like RDMA receive-slot reuse. Config formats and CLI surfaces should be expected to move between minor versions. ## Verdict OpenLake is attacking a genuine and expensive bottleneck — recomputed prefill is one of the largest sources of wasted GPU-seconds in production LLM serving — and it does so with an architecture whose every choice (io_uring, core pinning, GPUDirect, paced RDMA, erasure coding) is coherent with the sub-millisecond target rather than bolted on. The vLLM connector making it a config change rather than a migration is what makes it worth an afternoon of evaluation. Teams running multi-node GPU fleets with long, repetitive prompts and RDMA fabric already in place should benchmark it directly against their current prefix-caching setup. Teams on single nodes, commodity networking, or without the appetite for a v0.x dependency in the storage layer should watch it and revisit at 1.0.