Latest in AI
llamAmpere v0.6.2: Qwen 27B Quantization, VRAM and Runtime Requirements
A new low-bit Qwen derivative makes a smaller local model practical, but its memory estimates need careful reading.
llamAmpere released v0.6.2 on November 1, adding a 2.5-bit tensor format used by a new, roughly 2.3-bit-per-weight Qwen3.8-27B derivative. The appeal is smaller local inference on NVIDIA GPUs. The catch is that this file needs a specific runtime, and its published 12 GB configurations remain untested.
What llamAmpere v0.6.2 adds
The runtime release adds a fractional representation that packs two weights into five bits, alongside a scale value. This gives the quantizer another option between two-bit and three-bit storage instead of forcing every tensor into the same precision.
Its companion model, ATX AI2, comes from the refusal-reduced Swift 1.5 derivative of Qwen3.8-27B. It is a community model release, not a new official Qwen checkpoint. The file contains the text model and a multi-token prediction draft layer, but no vision tower.
The GGUF extension does not make this a drop-in llama.cpp model. Stock llama.cpp and llamAmpere v0.6.1.1 or earlier cannot load it. Anyone downloading the weights needs v0.6.2 or newer, rather than assuming an existing local application will understand the new tensor type.
File size is not the GPU memory requirement
The model card lists a 9.14 GB download. Its approximately 2.32 bits-per-weight figure excludes the token embedding table, which stays in system RAM.
The release notes put model weights, including the draft layer, at 7,416 MiB on the GPU, with another 1,288 MiB of embeddings in RAM. That leaves attention memory, working buffers and desktop usage still to account for.
| Configuration | What the release actually establishes |
|---|---|
| Linux, 24 GB RTX 30-series card | The published quick-start configuration was tested with a 204,800-token context setting |
| 12 GB card | A 204,800-token configuration is provided, but its roughly 11,000 MiB estimate comes from older measurements |
| 10 GB card | A 49,152-token configuration is provided as an estimate, not a test of this file |
The detailed release notes explicitly say the smaller-card commands have not been run on this model. Other GPU generations and Windows were also outside the stated release checks.
For someone choosing whether to download it, that distinction matters more than the headline bit count. A long conversation increases the key-value cache, the stored attention information used while generating replies. Fitting the weights is only the first part of fitting the workload.
What the coding result does and does not show
The author reports 84.3% on a 100-problem LiveCodeBench v6 evaluation run three times, using the strict grader. A modified grader that accepts alternative correct forms raises that to 88.3%. Those numbers should not be interchanged.
More importantly, the strict result is not statistically distinguishable from the author's previous best 2.3-bit build in this test. The model card publishes that limitation, alongside evaluation data. These are author-run evaluations, not independent replication.
The coding runs used a 48 GB card with two conversation slots and 262,144 tokens per slot. They therefore do not establish the same coding result on the proposed 12 GB setup.
Who should try it
This is most relevant to local-model developers comfortable building a Linux CUDA runtime and checking memory settings themselves. The server offers an OpenAI-compatible chat API, but compatibility at that interface does not remove the specialized backend requirement. The model inherits Swift Open License 1.0, so commercial users should review its terms.
The useful next test is reproducible smaller-card performance with realistic prompts. Until then, treat the 12 GB recipe as an experiment, not a verified buying recommendation.
I’m Paras. I work at the intersection of AI, engineering and distribution.
I help technical founders turn model releases and engineering work into clear stories developers want to read and share.