HOPE Expert Pruning: How to Shrink a Qwen MoE Model

HOPE helps shrink mixture-of-experts models by removing experts while accounting for how they work together. Amazon's November 9 walkthrough makes it worth a closer look for teams running Qwen models themselves, especially when model weights leave too little GPU memory for useful workloads.

The practical question is how to make a smaller checkpoint without quietly breaking the coding agent that depends on it. That requires a calibration dataset, a compatible model implementation and a comparison against the original checkpoint. A smaller file is only the beginning.

What HOPE changes about expert pruning

A mixture-of-experts model routes each token through a subset of its experts. That reduces active computation, but keeping the expert weights available still consumes memory. Pruning removes some of those weights permanently.

HOPE considers pairs of experts when choosing what to remove, rather than ranking every expert separately. The research paper evaluates this approach across three models and two calibration datasets. Its reported advantage is strongest at high pruning ratios and on agentic tasks. The paper dates to September; the November 9 publication is a practical walkthrough, not a new model release.

For an engineering team, the distinction matters because calibration becomes part of the deployment decision. You are choosing which capacity to retain for a workload. Treat the resulting checkpoint as a new artifact that needs its own acceptance tests.

Check model compatibility before downloading weights

The official HOPE repository documents testing with Transformers 5.2.0 on Qwen3 MoE, Qwen3-Next, Qwen3.5 MoE and GLM-4.5-Air. Its observer expects the stacked-experts weight layout. A model being described as MoE does not establish that this implementation can process it.

Start in an isolated environment and follow the repository's installation instructions, including a PyTorch build appropriate for your CUDA setup. Record the model revision and software versions before producing a checkpoint. Otherwise, a later library upgrade can make a failed reproduction difficult to diagnose.

Also budget for the original model during preparation. The benefit arrives after pruning; do not plan the calibration job around the smaller checkpoint you hope to produce. Confirm the source model's license separately from the pruning package's license.

Prepare calibration and evaluation data separately

Use calibration prompts that represent the work you expect the model to perform. For a coding assistant, that means more than short requests to write functions. Include the kinds of repository context, tool results and debugging exchanges that your application actually sends.

Keep a separate evaluation set that the pruning decision never sees. A useful split might include routine edits, unfamiliar code, broken tests and ambiguous instructions. These are suggested evaluation categories, not results from testing HOPE.

The important failure case is a checkpoint that looks good on familiar examples but loses reliability on the difficult work you needed the larger model for. Choose the acceptance criteria before choosing the pruning ratio. For a coding workflow, those criteria should include completed tasks and valid tool use, alongside memory consumption.

Run the three-stage workflow

The repository's command-line workflow collects observations, selects experts and writes a separate checkpoint. Its text input format accepts one prompt per line. The following commands illustrate that documented flow with Qwen3.5-35B-A3B; they have not been executed for this article.

hope calibrate \
  --model-path Qwen/Qwen3.5-35B-A3B \
  --prompts calibration_prompts.txt \
  --out-path observations.h5

hope solve \
  --obs-path observations.h5 \
  --prune-frac 0.25 \
  --out-path pruneset.json

hope prune \
  --model-path Qwen/Qwen3.5-35B-A3B \
  --pruneset-path pruneset.json \
  --out-path ./qwen35-pruned-25

The 25% setting here is an evaluation starting point, not a recommended production setting. Keep the source weights intact and give each candidate its own output directory. Save the calibration file and selection output with the candidate so you can explain how it was produced.

Afterward, run the original and pruned checkpoints through the same serving configuration and evaluation harness. Record peak memory at a fixed context length and concurrency, task completion, malformed outputs and latency. Change one variable at a time: changing quantization, serving software and pruning together makes any regression much harder to attribute.

How much quality should you expect to keep?

Amazon's November 9 explanation reports a SWE-Bench Pro score of 42.3 for Qwen3.5-35B-A3B at 50% pruning, compared with 41.8 for the unpruned model and 36.2 for REAP. Those are the authors' results in their setup, not a guarantee for another agent harness.

The same explanation acknowledges that simpler methods can be competitive at lower pruning rates. HOPE also works layer by layer and models pairwise interactions, leaving broader interactions outside its practical formulation.

A 50% expert reduction should not become a promise of 50% lower serving cost. Your deployment still has other memory requirements, and the business result depends on whether the smaller checkpoint changes the hardware or concurrency you can actually use. Measure that outcome instead of estimating it from the pruning percentage.

When HOPE is worth the work

HOPE is a sensible candidate when you control inference, weights are a material constraint and you can evaluate a replacement checkpoint. Teams that only call a hosted API cannot apply this workflow to a provider's hidden weights.

The strongest reason to prune is a concrete deployment improvement: a checkpoint that fits the intended hardware or leaves enough capacity for the workload, while still meeting the quality threshold. If the only result is a smaller download, the engineering work has not yet paid off.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams turn model research and engineering decisions into clear stories developers can understand and use.

Book a 30-minute call.