# Resin Explained: Rust Scheduling and KV Cache for Custom LLM Inference

Resin separates inference scheduling from model kernels, giving developers a smaller starting point for experiments with new models and hardware.

By Paras Madan | 2026-10-26


Eric Zhang introduced Resin on October 26, an experimental library for building LLM inference systems. It provides a Rust scheduler and cache-management core while leaving model execution to separately written kernels. The audience is developers experimenting with inference internals, not teams looking for a production replacement for vLLM.

That distinction matters because serving a model requires more than making its matrix operations fast. Requests arrive at different times, prompts share prefixes, memory fills up, and generated sequences finish at different lengths. Someone still has to manage all of that around the forward pass.

## What Resin actually provides

The [Resin repository](https://github.com/ekzhang/resin) describes a headless engine with optional HTTP serving and Prometheus metrics. The core can be installed from GitHub with Python 3.11 or later and a Rust toolchain. CUDA-oriented implementations and Torch reference implementations live in separate model-runner packages.

Resin is explicitly experimental. Its own documentation recommends SGLang or vLLM for people who need a production inference engine. It is better understood as a place to develop and inspect an engine than as a ready-made service with an operational guarantee.

For someone asking “Resin vs vLLM,” that is the first answer. The projects are not interchangeable choices at the same level of maturity. Resin exposes building blocks for people who want to change how inference works.

## How the scheduler and model kernels fit together

The [architecture documentation](https://github.com/ekzhang/resin/blob/main/docs/architecture.md) draws a firm boundary. The Rust engine owns request admission, priorities, batching, prompt processing in chunks and cache allocation. It also tracks request progress and completion.

The model runner owns weights, device memory and the actual computations. It takes a frozen batch plan from the engine and converts that plan into the metadata its kernels need. Attention, sampling, collectives and CUDA graphs belong on this side of the boundary.

After execution, the runner returns results for the engine to validate and commit. **Choosing the next batch and calculating that batch are separate responsibilities.**

Consider a developer trying a different attention implementation for one model. Ideally, that experiment should not also require rewriting how waiting requests are scheduled or how completed requests release memory. Resin's boundary gives that work somewhere more specific to live, although the integration still has to be correct.

## Why cache ownership deserves its own layer

The key-value cache stores intermediate attention information so a model does not recalculate the entire preceding sequence for every generated token. Keeping that cache useful across many requests is a separate systems problem from implementing attention itself.

Resin's architecture supports full-attention, sliding-window and recurrent retention, along with token-exact prefix reuse. The engine manages page ownership and leases rather than leaving every model runner to invent its own memory-lifetime rules.

This makes the design interesting beyond the choice of Rust. If a kernel experiment produces a wrong answer, developers need to distinguish numerical errors from scheduling or state-reuse errors. Keeping ownership explicit gives them a more manageable boundary to inspect. It does not prove that the implementation is bug-free.

## What Resin's benchmarks do and do not show

The creator reports competitive results from model-specific implementations, but those are project-reported measurements, not an independent production evaluation.

The more useful starting point is Resin's [benchmark methodology](https://github.com/ekzhang/resin/blob/main/docs/benchmarks.md). It separates prefill, when the model processes the prompt, from decode, when it generates subsequent tokens. The default runs include warmup and measured iterations, with different prompt lengths and decode batch sizes.

Those measurements exclude startup, weight loading, capture and request admission. Decode measurements also exclude the preceding prefill. That makes them useful for investigating execution performance, but **they are not a complete user-facing latency measurement**.

An application team would still need to measure its own request mix, cold starts, queueing and response-time targets. A fast controlled batch cannot establish those outcomes by itself.

## Where developers should be careful

There is a practical cost warning in the example deployment, too. The repository's Modal example uses `requires_proxy_auth=False`. Anyone who has the endpoint URL can submit requests and incur GPU usage. A runnable demo should not be mistaken for a deployment with suitable access controls.

The sensible audience is inference researchers, kernel developers and engineers exploring support for new hardware or architectures. Developers who only need a reliable model endpoint have a different problem and should take the project's production warning seriously.

Resin's useful proposition is the narrower surface for experimentation. If the scheduling and cache layer can remain stable while model-specific execution changes, more of an engineer's time can go into the part they actually wanted to improve. The next thing to watch is whether that separation holds across genuinely different models and hardware, not just a favorable throughput chart.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders explain complex AI products and turn engineering work into launch stories that developers can understand and share.

[Book a 30-minute call](https://cal.com/paras-madan/30min).