Langfuse Experiment Comparison: Baselines, Charts and Regression Review

Langfuse updated its experiment comparison view on October 19 with customizable score charts, baseline-relative summaries and easier item-level review. The release helps teams compare changes to an AI application, then inspect the individual answers behind an aggregate score before deciding whether the change is worth shipping.

The announcement builds on Langfuse's existing experiment workflows. It does not introduce a new model or evaluator. The change is in how results from different runs are presented and reviewed, which matters because a single average can hide the failure that makes a release unacceptable.

What changed in Langfuse experiments

The new comparison view lets users customize charts across experiment runs and see summary statistics relative to a chosen baseline. Instead of treating every run as an isolated result, a team can use its currently accepted version as the reference point.

For closer inspection, item-level score thresholds help narrow the cases to review. Metric columns and row heights can be adjusted, and keyboard navigation makes it easier to move through examples. Human scores, corrected outputs and comments can be added in context.

Langfuse's instructions are to open Experiments and select two or more runs. The announcement does not specify a new price, a plan restriction or an exact minimum version that self-hosted installations need. Those details should be checked against the deployment being used rather than inferred from a nearby software release number.

How to compare an experiment against a baseline

The comparison documentation covers runs using Langfuse datasets as well as local data. The important setup question is whether the runs represent a meaningful comparison.

Suppose a support assistant currently uses one model and the team wants to try a cheaper option. Run both versions against the same representative cases, using the same evaluator configuration. Record the model, prompt and application version so a result can be traced back to the actual change.

This is easy to undermine accidentally. If the candidate gets an updated retrieval system, a different prompt and a new evaluator at the same time, the resulting score does not isolate the effect of the model. The comparison can still be useful as a test of the whole application, but it cannot tell you which individual change caused the result.

Selecting a baseline in the interface also does not set a deployment policy. A team still needs to decide what constitutes a regression and whether that decision is enforced in its release process. The view helps inspect evidence; it does not automatically turn an experiment into a CI gate.

Why the average score is not enough

Imagine both versions of the support assistant achieve the same average score. The cheaper candidate improves several routine answers but starts giving the wrong response to a small group of cancellation questions. The summary can remain unchanged while the product becomes worse in an important part of the workflow.

This is where item-level review earns its time. Look at cases that newly fall below an acceptable score, rather than only sorting runs by their overall average. Review the original input, the answer and the trace together before treating a low score as a model failure.

A failed case may come from retrieval returning the wrong document, a tool call returning incomplete data or the evaluator itself failing. These require different fixes. Missing scores and evaluator errors should remain visible as unresolved results, rather than quietly being counted as passes.

Human annotations are useful for these disagreements. A reviewer can record why an answer should pass or fail and leave a corrected output. If the evaluator's criteria change as a result, rescore both the baseline and candidate so the next comparison uses the same standard.

Comparing model cost without losing the product goal

Langfuse's cost-optimization guidance connects evaluation with choices such as model selection, token use and the number of model calls. A cheaper token price is only one part of the calculation.

For the same support assistant, a lower-priced model could produce a correct answer in one call, or need a second attempt and a longer retrieval sequence. The useful cost comparison is the complete application run under a defined quality requirement. Evaluation calls themselves also add cost, so a test budget should include them.

The updated charts and baseline summaries make this tradeoff easier to examine, but they do not choose the quality threshold. A team may accept a small change on a low-stakes formatting score while rejecting any new failures on a critical workflow. That judgment needs to be explicit before an attractive cost reduction becomes a release decision.

What to do with the comparison

Use the aggregate view to find promising candidates, then review the cases most likely to change the decision. Keep the baseline stable, distinguish errors from missing evidence and preserve enough version information to reproduce the run.

The value of this release is that it reduces the distance between seeing a score and understanding what produced it. Better comparison tools are useful when they help a team explain exactly which behavior improved, which behavior regressed and why the resulting tradeoff is acceptable for its users.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI and developer-tool teams explain evaluation workflows, technical decisions and product launches through clear stories that developers and founders read and share.

Book a 30-minute call.