PARAS MADAN
All writing

Latest in AI

LangSmith Coding Agent ROI: Check Attribution Before Comparing Costs

Set up LangSmith coding-agent outcomes, diagnose missing PR attribution and compare spending without mistaking merged pull requests for proven business returns.

A lower cost per merged pull request sounds like a good reason to switch coding models. It can also mean that this week's work was easier, expensive investigations were left outside the comparison, or part of the tracing setup is missing.

LangSmith's November 18 release connects coding-agent spending to GitHub pull requests and GitLab merge requests. It supports relationships where several threads contribute to one PR or one thread contributes to several PRs. The public beta also separates unattributed spending, which can include useful investigations and reviews. The announcement describes the new reporting.

This gives engineering teams a better starting point for cost analysis. It does not establish that an agent generated a financial return. Here is the pilot I would run before using the dashboard to change a team budget.

Establish what the dashboard can see

The Coding agent type requires an agent-based workspace on LangSmith Cloud, not self-hosted LangSmith. Send traces to an agent with that type, then connect GitHub or GitLab. Repository access and connection permissions must be available to the person doing setup.

Linking uses repository and source-branch metadata when the provider reports PR activity. Pull requests merged before connection do not appear. These are important limits in the implementation guide, not reasons to assume older work produced nothing.

Record the connection date alongside the start of your pilot. I would not put an incomplete historical period beside a fully instrumented week and label the difference an improvement.

Check one real PR end to end

Choose a small, ordinary change rather than the team's most impressive demonstration. Have the developer identify its working branch, coding-agent conversation and resulting PR. Check that the linked threads describe that change.

If the PR has no threads, inspect the integration before judging the model. Copilot records Git attribution only through its Local harness in a folder with a Git remote. Deep Agents Code can have Git attribution disabled. The work-type chart separately requires the Coding Work Type evaluator.

For a custom integration, the metadata contract specifies a stable thread_id and identity fields including ls_integration and ls_trace_schema_version. Git fields such as ls_git_repo, ls_git_repo_url and ls_git_branch are required when the runtime knows them. Do not invent missing values just to make a chart populate.

Validate the cost before the ratio

There is a second completeness check. LangSmith's cost-tracking documentation says child runs need thread metadata to contribute to thread-level token and cost totals. Automatic cost calculation also depends on token counts, model/provider identity and model pricing; additional costs can be submitted manually.

My proposed audit is to choose one conversation and compare its visible child calls with the thread total. Record which expenses your reporting includes. A model-usage estimate and an all-in cost that includes hosting, subscriptions and human review are different measurements.

For illustration only, a hypothetical $60 of attributed model spending divided by six merged PRs is $10 per PR. That arithmetic says nothing about review hours, defects or revenue. Keep the denominator and the cost definition written next to the number.

Keep investigations in the review

I would classify a small sample of unattributed threads manually before deciding what to cut. Separate a useful investigation, abandoned work, unrelated work and a suspected instrumentation failure. Give each category a concrete example that the team can recognize.

An investigation that rules out a risky migration may be valuable precisely because it ends without a code change. Conversely, creating a PR is not evidence that a change was necessary. Ask the responsible engineer what decision or deliverable the session supported.

Compare similar work, then change one thing

For the first comparison, keep the repository and task category consistent. Record model, harness and relevant workflow changes. Avoid switching the model, rewriting instructions and introducing a new review process together if you want to understand what caused a difference.

Pair your cost comparison with review effort and a follow-up check for reverts or defects. Track those separately rather than treating the dashboard as an automatic ROI calculation.

The useful outcome is a decision you can explain: which workflow to keep, what to investigate and which missing data prevents a conclusion. A dashboard should make that conversation easier, not replace it.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams explain technical products through clear documentation, practical guides and developer-focused content.

Book a 30-minute call.