PARAS MADAN
All writing

Latest in AI

Inkling 2 Mini: Run It Locally Without Assuming a 1M-Token Budget

Choose a supported Inkling 2 Mini runtime, verify the checkpoint format, and size context and concurrency separately from the model download.

Thinking Machines Lab released Inkling 2 Mini on November 20. This is the new 26.4B model, not the earlier Inkling 2 Small. The publisher shows it running on a laptop and documents a model context of up to one million tokens. Those are different claims: being able to load the model does not mean a local setup can run its entire context window.

Start with the runtime and memory budget you actually have, then increase the context only when your workload calls for it.

Start with the Mac route if it fits

The publisher supports the Ollama route on Macs with at least 32 GB of memory. The Ollama registry lists downloads around 20 GB and provides this starting command:

ollama run inkling-2-mini

Use a current Ollama build that supports the model. Make space for the download and leave memory for the application, the operating system and the conversation. The download size is not the memory required for your longest prompt.

Use a short text prompt for a first run and then a small image with a question that cannot be answered from the text alone. That separates a working text decoder from a working vision path. If you are trying it for coding, follow with a small local repository task and measure how long the full task takes, including any tool calls, rather than looking only at tokens per second.

Choose a GGUF that matches the runner

The Hugging Face integration guide documents an official ggml-org GGUF for llama.cpp. Start with the artifact recommended for that runtime and record the exact build and file revision. An arbitrary quantization with the same model name is not necessarily interchangeable.

For example, on November 21 llama.cpp merged a fix for the scale tensors used by certain NVFP4 weights during repeated passes. The change does not affect ordinary ggml quantizations, according to the pull request. For an NVFP4 GGUF, confirm that the build you use includes the fix before interpreting surprisingly poor output.

A failed load or broken short answer is not a reason to buy more memory yet. First verify that the runner supports your specific checkpoint. Retest with a documented model and known input before mixing a new quantization, a different runtime and a long prompt.

Understand what looping costs

The lab distinguishes 3.4B active parameters from about 5.4B weight activations per token after accounting for repeated computation. The middle layers reuse their weights across three passes. Reused weights do not mean those passes are computationally free.

The vLLM serving recipe explains that the repeated passes keep their own attention and convolution state. The weights may fit while a long session or several concurrent sessions put pressure on memory. Budget for those independently from the checkpoint.

For example, a coding agent that reads a few files and returns a patch does not need the same allowance as an assistant that ingests a large document collection in one request. Start with the shortest cap that covers real requests plus output headroom. Raise it separately from concurrency so you can see which change causes latency or memory pressure.

Use the recipe for your GPU, not its biggest number

vLLM currently points to a dedicated Inkling 2 Mini image or a compatible nightly. Its reference recipe describes a 1,048,576-token context at tensor parallel size one on GB200, but a 262,144-token cap on H100 and DGX Spark. Select your hardware profile and examine the rendered settings rather than copying the GB200 example.

For a server evaluation, track peak memory, time to first visible answer and completed-request latency at several prompt lengths. Turn on speculative decoding only as a separate comparison: the recipe warns that acceptance depends on actual prompts and sampling temperature.

If the application uses tools, test that parser path independently. A correctly decoded chat answer says nothing about whether a tool call reaches your application with valid arguments. Keep the first tests read-only and compare how many tasks actually complete, not just how fast text appears.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams explain launches and deployment tradeoffs through technical writing developers can use.

Book a 30-minute call