Latest in AI
Running OVH Olala locally: choose the runtime before the weights
Olala needs vendor runtime forks, different M1 flags and careful client testing. Here is how to choose a format without confusing file size with memory.
OVH's Olala release gives developers another local model to evaluate. The first decision, however, is not which quantization to download. It is which runtime can actually load the architecture and serve your application correctly.
The official model card dates the release to November 19 and lists Apache 2.0 licensing, 6.8 billion total parameters and roughly one billion active per token. It combines Mamba and attention layers with a mixture-of-experts design. Supported context is 32K tokens, counting input and output together. A 128K version is described as forthcoming, not available in this release.
That makes this a setup and integration decision before it becomes a benchmark comparison.
Start with a runtime, not an app's model menu
Olala's launch instructions require vendor forks. The main card's NVIDIA serving route uses OVH's vLLM fork, with Python 3.12 and a 24GB-or-larger GPU recommendation. Its listed test hardware includes L4 and L40S.
Do not treat a familiar download button as evidence that your existing installation supports the model. Pick one runtime, record its exact revision, and get a short request working before adding your application framework.
I would keep the initial environment separate from a working local-model setup. A fresh environment makes dependency conflicts easier to identify and gives you a clean rollback. Save the model revision alongside the runtime revision so a later comparison changes one thing at a time.
GGUF: smaller files are not automatically faster
The GGUF card requires OVH's llama.cpp fork. Upstream llama.cpp, Ollama and LM Studio do not yet support this architecture.
Published file sizes include 13.6GB for BF16, 6.9GB for q8_0-safe2, 6.3GB for q6_k and 6.0GB for q5_k_m. The card recommends q6_k for a 16GB Mac. Its own M5 measurements put q6_k ahead of the smaller q5_k_m for generation speed. That is a hardware-specific result, not a promise for another laptop.
On Apple M1, the GGUF instructions specify this performance flag before starting the CLI or server:
export OLALA_NO_FUSED_MOE=1
The practical lesson is to compare candidate formats on your workload rather than rank them by file size. For a document assistant, test both a short question and a long retrieved passage. A format that starts quickly on a tiny prompt may still disappoint when every request includes several pages.
MLX: M1 needs a different flag
The MLX card requires OVH's mlx-engine fork, not stock mlx-lm or LM Studio. Its q5e6 and q4e8 folders each list 4.9GB of weights.
On M1, set the following before Python or the server to avoid the documented first-token crash:
export OLALA_MOE_V3=0
This is not the GGUF setting. Label any launch scripts by runtime rather than collecting both flags in a generic shell profile.
Budget memory beyond the download
A weight-file size is not a total-memory guarantee. Leave room for the operating system, runtime allocations, context-related state and other applications.
For an initial trial, I would use one request at a time and a context budget comfortably below the advertised ceiling. Measure peak process memory during a representative long request, not just after loading the weights.
Then increase context or concurrency separately. If you change both together and hit a failure, you have made the cause harder to isolate. Also reserve output space: a context limit covering both directions is not an input-only allowance.
Test the client contract before building around it
MLX's server is streaming-only and requires temperature and top_k. The OpenAI Python client passes top_k through extra_body. Several familiar controls, including seed and frequency penalties, are rejected. Reaching max_tokens ends the stream with an error event rather than a normal length finish reason; previously streamed text remains valid.
Those differences deserve explicit tests. Send one ordinary completion, one deliberately constrained output and one tool request. Check what your UI shows when text arrives before an error. Make sure it does not silently present a truncated answer as complete.
For a local document assistant, retain the partial answer for inspection but mark the request incomplete. Retry only under a defined policy. Otherwise a client can turn a token-limit failure into repeated work while hiding the reason from its user.
Decide whether it fits your application
Keep a small evaluation set containing your actual documents, expected answers and unacceptable mistakes. Compare correctness, time to first useful output, completion time and peak memory together.
A model that loads is ready for evaluation, not automatically ready to replace your existing backend. Advance only when the runtime, output handling and task quality all pass the same reproducible checks.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help AI teams turn complex releases into useful technical writing that developers can follow.