# How to Estimate Fine-Tuning Time Beyond GPU FLOPs

A new study of production fine-tuning jobs shows why scheduling needs more than a compute estimate. Start by measuring the stages your deadline actually includes.

By Paras Madan | 2026-11-19


Dharma-AI's November 18 [fine-tuning time study](https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time) examines 272 supervised fine-tuning runs on shared H200 infrastructure. Its CatBoost predictor reports a 1.71-hour mean absolute error in five-fold cross-validation, compared with 12.41 hours for its best listed FLOP baseline. The full paper is still forthcoming.

The practical lesson is not to borrow that predictor's accuracy for your cluster. It is to stop treating training-loop compute as the whole job. If someone needs a usable checkpoint by tomorrow morning, your estimate should describe when that checkpoint will actually be ready.

I would begin with a timing ledger, not another model.

## Define the clock before estimating it

The study measures from GPU allocation to job completion and excludes queue time. Its runs include setup, loading, tokenization, evaluation and checkpoint transfers. They cover LoRA and full fine-tuning, models up to 15 billion parameters, and one, two or four H200s.

For your own scheduler, write down the start and finish events. For a delivery deadline, I would record submission, allocation, first training step, last training step and verified artifact availability separately.

That distinction prevents an awkward reporting error: calling a job finished because optimization stopped while the final artifact is still uploading. It also keeps a long queue from being blamed on slow kernels.

Use stable identifiers so the configuration, timing events and output checkpoint can be joined later. If you cannot reconstruct which run produced a duration, it is not useful training data for a predictor.

## Record the settings that create extra work

Alongside model size and training tokens, save the evaluation schedule, evaluation-set size, checkpoint schedule, checkpoint contents and allocated hardware. Include the software version and whether data and model files were already cached.

These are proposed fields for your own measurement plan, not a claim that I have deployed the study's system. Choose fields your platform can collect reliably rather than building a sprawling schema nobody maintains.

For an illustrative calculation, a ten-minute evaluation repeated 12 times adds two hours. A five-minute checkpoint transfer repeated six times adds another half-hour if those transfers block progress. Neither number is a measured result from the study. They simply show why apparently small configuration choices deserve a place in a deadline estimate.

Do not remove useful evaluation or recovery checkpoints just to improve the displayed completion time. First establish whether the schedule serves a real quality or recovery requirement.

## Build a baseline you can explain

Start with comparable historical jobs. Group them by training regime, hardware and workload range, then record both the typical duration and the spread. A clearly qualified estimate from a few similar runs is more useful than a precise-looking number drawn from an unrelated setup.

Keep a compute estimate too. It can serve as a reference against which you investigate extra elapsed time, rather than an answer that must explain every stage.

When a run misses its estimate, classify the miss. Was there an unusual download, a storage delay, a larger evaluation workload or a changed software stack? Retain exceptional runs with an explanation instead of silently deleting inconvenient evidence.

## Test predictions on the jobs that arrive next

The reported predictor is calibrated to H200 SFT jobs, not arbitrary GPUs or other training objectives. Its errors widen on runs longer than 60 hours, where the dataset contains only 13 examples. Feature importance is not evidence that changing a setting will cause the predicted improvement.

Before putting any learned estimate into scheduling, I would compare its predictions with subsequently completed jobs. That checks the situation you actually care about: predicting work that has not happened yet.

Track underestimation separately from overestimation. An average absolute error does not tell you whether a critical overnight job will finish before a morning handoff. Choose a planning buffer from your own observed misses and explain what it covers.

## Keep scheduling confidence separate from model quality

A job finishing on time does not mean its checkpoint is good. Keep the acceptance evaluation as a separate gate, with an owner and a fallback if it fails.

My preference is a simple report: expected artifact-ready time, assumptions, likely delay range and the next point at which the estimate will be updated. That is actionable for a team waiting on training. A theoretical runtime with no account of the surrounding work is not.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI infrastructure teams turn engineering research into clear technical articles and developer guides.

[Book a 30-minute call](https://cal.com/paras-madan/30min).
