Hugging Face Decision Pipeline: Setup, System One API and 422 Errors

Hugging Face's November 10 decision-model integration gives developers a Transformers pipeline and a /v1/systemone endpoint for answering constrained questions. Instead of generating JSON and parsing the answer, an application supplies allowed outcomes and receives their probabilities. The useful change is a clearer contract between model inference and application logic.

The announcement covers a broader ecosystem, including llama.cpp. This guide focuses on getting the Transformers interface working and separating request errors from prediction errors. The examples below are documentation-based implementation examples, not results from a model test.

Install the version that contains the decision pipeline

The documentation currently describes the Transformers main branch, and the decision API is experimental. A normal package upgrade should not be treated as proof that the installed release contains it. Start in an isolated environment with a PyTorch installation appropriate for your machine, then install the official source with serving dependencies:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install "transformers[serving] @ git+https://github.com/huggingface/transformers.git"
python -c "import transformers; from transformers import DecisionPipeline; print(transformers.__version__)"
transformers serve

These are Unix-shell commands and require Git. Follow the source installation instructions for platform-specific setup. If the import fails, check which Python environment is running before investigating your model or request body.

For a reproducible deployment, pin the source revision after validating it. An experimental API tracking a moving branch is a poor dependency for an unattended production update.

Send a small, explicit decision request

The following example classifies product feedback rather than asking the model to write a response. Save it as decision-request.json:

{
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "state": "The CSV export fails whenever my report contains more than 10,000 rows. Smaller reports export correctly.",
  "questions": {
    "feedback_type": {
      "type": "choice",
      "instructions": "Classify the main issue described in this product feedback.",
      "criteria": {
        "bug": "An existing feature fails or behaves incorrectly.",
        "feature_request": "The user requests a capability that does not exist.",
        "pricing": "The user asks about price, billing or subscription terms."
      }
    },
    "has_reproduction_detail": {
      "type": "noul",
      "instructions": "Does the feedback describe a condition under which the problem occurs?"
    }
  }
}

Then send it to the local server:

curl --fail-with-body http://localhost:8000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary @decision-request.json

The serving documentation uses this small Qwen model as an example. It is an integration starting point, not a recommendation based on measured classification quality. First use can require a model download and sufficient local memory.

Keep the question names stable. They are application-facing identifiers, whereas the instructions and criteria define the task the model must perform. Changing “bug” from a broken feature to any negative feedback changes the classifier's meaning even if the JSON schema stays identical.

Read the response without treating it as generated chat

HTTP responses put results under answers, keyed by your question names. The Python pipeline("decision", ...) interface returns the question-keyed dictionary directly. Code written for one wrapper can therefore fail against the other even when inference succeeds.

The DecisionPipeline documentation explains that compatible chat models score answer labels from their next-token logits. Native decision models can score options directly. The implementation performs one forward pass per prompt, which does not necessarily mean one pass for an entire multi-question request.

For choice, inspect the selected option and its probability distribution. For noul, the result expresses a true/false decision. The third question type, score, uses ordered criteria from lowest to highest and returns an expected index. For example, probabilities of 0.1, 0.2 and 0.7 across three levels yield a score of 1.6. That is a weighted value, not the name of a fourth category.

Confidence and the highest probability are not interchangeable. The announcement describes choice confidence as a normalization against a uniform distribution. With three choices and a highest probability of 0.8, that calculation gives 0.7. Neither number establishes an 80% real-world accuracy rate without evaluation on representative examples.

Diagnose request failures before changing models

A connection failure means there is no reachable HTTP service at that address. A 404 suggests the wrong route or a server build without the endpoint. Neither tells you whether the model can classify your feedback.

A 422 is different: the documented endpoint validates the request before loading the model. Check lowercase question types, the plural instructions field, and the shape of criteria. Choice criteria are a mapping; score criteria are an ordered list. The current interface allows up to 64 questions and, with its default A–Z labels, up to 26 choice options or score levels.

Once the request is accepted, treat download errors, memory failures and unexpected classifications as separate problems. Rewriting the taxonomy will not fix insufficient memory, and reinstalling the package will not resolve overlapping labels.

For the worked example, a useful test set would vary the failure condition, include a genuine request for a new export format, and include an ambiguous complaint. Agree on expected categories before inspecting predictions. Otherwise it is easy to keep changing labels until the first demonstration looks correct.

Moving the request to llama.cpp

llama.cpp already documented System One support in its October decision-model guide. This is not a second launch today. The shared text request shape makes it worth evaluating as an alternative runtime, but use the server address printed by your installed build rather than copying a port from an older example.

A successful request against two backends establishes compatibility, not equivalent predictions. Keep a small contract test for response fields and a separate evaluation set for decisions. Version the model, question wording and application thresholds together. That is how a simpler inference interface becomes a maintainable product component rather than another source of silent behavior changes.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams turn implementation details and product decisions into clear writing that developers can use.

Book a 30-minute call