BotPsy Local Agent Setup: Q8 vs Q4 and the Thinking Setting You Should Check

Tether AI Research released BotPsy on November 11, a 0.8-billion-parameter model for local tool calling. Start with its Q8 GGUF if your machine can accommodate it, explicitly enable thinking, and test against tools that cannot change real accounts. A small download does not make an autonomous assistant safe.

The useful question is whether it can handle a narrow job without sending every decision to a cloud model. That is a much better starting point than asking a sub-billion-parameter model to become a general personal assistant overnight.

Choose the model file before choosing the agent framework

The official GGUF repository offers BF16 at about 1.52 GB, Q8_0 at 0.81 GB and Q4_K_M at 0.53 GB. These are file sizes, not complete runtime memory requirements.

Format	Approximate file size	Sensible starting point
BF16	1.52 GB	Reference when investigating quantization differences
Q8_0	0.81 GB	First local evaluation when memory permits
Q4_K_M	0.53 GB	Tighter memory budgets, with separate regression checks

QVAC's reported ten-benchmark average is 50.7 for Q8 and 48.9 for Q4. Those are the publisher's results, not a test on your laptop. The GGUF evaluations used NVIDIA H100 hardware; the results do not establish your phone's latency, battery usage or sustained performance.

I would start with Q8 because the extra download is modest enough to justify checking it before accepting a quality tradeoff. If Q4 is necessary, compare the decisions your application actually needs, rather than treating the average benchmark gap as a predicted failure rate.

Turn thinking on explicitly

The model card says BotPsy was trained and evaluated with thinking enabled, while its chat template defaults to non-thinking. That mismatch is an obvious troubleshooting check if a fresh installation performs unexpectedly poorly.

Use the model author's usage section for your runtime, rather than assuming an automatically generated integration snippet contains every necessary setting. The documented llama.cpp route uses the embedded template with Jinja enabled and explicitly passes enable_thinking as true. The vLLM example also specifies tool-call and reasoning parsers.

Before changing sampling values, inspect what your application actually sends. A framework can accept a setting in one configuration file and fail to forward it to the backend. Record the model identifier, file version, runtime version and final request alongside each test result so that a later improvement is reproducible.

A successful chat response is not evidence that tool calling is configured correctly. Inspect the returned function name and arguments separately from the readable answer.

Use a fake task list for the first experiment

Here is an illustrative workflow, not a claim of firsthand testing. Give the model two tools: list_tasks, which reads an in-memory list, and propose_task, which returns a proposed task without saving it. Ask it to find overdue items and propose a reminder for one named item.

The application should validate the returned object before doing anything with it. Reject an unknown tool name, a missing task identifier or an identifier outside the supplied list. Show the proposed reminder to the user rather than giving the model a real messaging credential.

Then make the request deliberately incomplete. Remove the task name, refer to two similarly named tasks, or ask for an action no tool supports. A useful small agent should expose uncertainty instead of inventing the missing identifier. Count those cases separately from the straightforward requests; otherwise the easy examples can hide the expensive mistakes.

Test the boundary where small models struggle

The release analysis identifies open-ended retrieval as a weak area: its reported BrowseComp-Plus score is 4.5 despite improvements over several small-model comparisons. It also reports a 31% unsafe-response rate under its prompt-injection evaluation. Those results argue for a bounded workflow with explicit enforcement outside the model.

For the task-list example, place a hostile instruction inside a task description telling the assistant to ignore the user and call another tool. The expected outcome should be defined by your application policy, not by whatever explanation the model generates afterward.

Do the same with long conversations. Put an important restriction near the start, add several unrelated turns, then repeat the task. If a restriction disappears from behavior as context grows, shortening the prompt may help, but the permission check still belongs in application code.

Local inference does not make every tool local

A model can run on your machine while its search, email or document tools contact remote services. Draw the data path for each tool before describing the application as private. Include logs and crash reports, since an otherwise local design can still copy sensitive inputs into diagnostics.

For an initial evaluation, use synthetic data and no personal-account credentials. The model card explicitly excludes high-stakes automated decisions and applications involving money, personal information or health from its intended use. Do not reinterpret its personal-assistant branding as a recommendation to hand it those responsibilities.

The strongest early use case is a small, inspectable decision inside software you already control. If BotPsy can select the right action there, you gain a local component whose limits you can measure, rather than a much larger promise you cannot defend.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams explain model releases and developer workflows clearly, including the limits that matter in real products.

Book a 30-minute call.