PARAS MADAN
All writing

Latest in AI

Hugging Face Jobs for GitHub CI: Fix Cache Misses and Queued Runners Separately

A practical migration guide for AI projects using HF Jobs and Buckets, including dispatcher restarts, cache ownership and a small rollout plan.

A slow AI build can spend most of its time waiting for a runner, rebuilding unchanged objects, or installing dependencies. Moving it to a GPU machine will not necessarily fix any of those problems.

The llama.cpp team’s November 16 CI write-up describes a useful separation. Pull requests run a selected set of checks, non-release compiler caches live in Hugging Face Buckets, and many Linux x64 jobs run on Hugging Face Jobs. GitHub Actions still coordinates the workflow.

For another AI project, the useful question is which part to adopt first. I would start with one slow build, keep the old workflow available, and measure where its time goes before changing the whole matrix.

Separate queue time from compilation time

Record five intervals for the same commit: waiting for a runner, starting its environment, restoring the cache, compiling, and running tests. Keep the hardware, compiler flags and test selection alongside those timings.

A long queue points toward runner capacity or dispatch. A fast start followed by a cold compilation points toward cache availability or compatibility. Repeated package installation points toward the runner image. These are different investigations, even when all three appear as a slow green check.

Try a second unchanged build before drawing conclusions. Then change one source file and repeat. This gives you a useful baseline for deciding whether shared cache storage is helping, without claiming that a warm-cache demonstration represents every pull request.

Make the cache publisher explicit

In llama.cpp’s setup, PR builds consume a shared compiler-cache snapshot. Eligible builds on master update it. The bucket removes the dependency on a previous runner’s local disk; it does not make every cached object reusable under different build settings.

For your pilot, write down the cache key, who can update it, and what happens when its download fails. Include the backend, architecture and relevant toolchain configuration in that review. A cache should accelerate a valid build, not become a prerequisite for building it correctly.

Keep an uncached run in the comparison. If retrieving a large archive costs more time than it saves for your smallest job, moving that job’s cache is not the first optimization to ship.

Treat the runner label as the last setup step

The jobs-actions project is explicitly beta. It bridges GitHub’s workflow to an ephemeral runner inside an HF Job, with labels such as hf-jobs-cpu-upgrade and hf-jobs-t4-small selecting compute. Changing a label does not create the dispatcher or configure its GitHub App.

Follow the official migration walkthrough for that setup. It warns that a sleeping cpu-basic dispatcher can miss a webhook while waking, leaving work queued. It also explains why a bare Ubuntu image may need dependencies that GitHub’s hosted image already contains.

Start with a harmless check that prints the toolchain version and runs a small test. Verify that its logs and final status appear in GitHub before moving an expensive accelerator job. Include dispatcher uptime, image preparation and cache storage in the cost comparison, rather than comparing only the test’s execution time.

Diagnose a stuck queue before rerunning everything

The dispatcher documentation identifies an important restart limit: tracking lives in memory and requires one dispatcher process. After a restart, previously queued work needs a new webhook or rerun to re-enter tracking. The reconciler does not discover events it never received.

When a job stays queued, inspect webhook delivery, dispatcher logs, repository eligibility and the requested runner label first. Then check whether an HF Job exists and whether its runner registered. Do not assume a new container and the pending GitHub job have successfully connected.

The dispatcher also distinguishes the idle-runner deadline from the overall JOB_TIMEOUT. An idle timeout is not your build timeout. Configure and test both deliberately, including a failed bootstrap, before expanding the pilot.

Keep untrusted PRs out of the first experiment

GitHub’s self-hosted runner security guidance warns that untrusted workflow code can compromise its execution environment and expose available credentials. An ephemeral runner reduces persistence; it does not make arbitrary code trustworthy.

My starting point would be trusted internal changes, minimal workflow permissions, and no production credentials. Review public-fork handling separately. A faster check is not a successful migration if it changes who can reach your secrets.

Ship the pilot when you can explain both its speed and its failure behaviour. Keep the old runner route ready until cache misses, dispatcher restarts and cancelled jobs have predictable outcomes.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help infrastructure teams explain technical trade-offs through practical guides developers can use.

Book a 30-minute call