PARAS MADAN
All writing

Latest in AI

Genie ZeroOps Public Beta: Start With One Pipeline and a Reviewed Fix

Set up Databricks Genie ZeroOps with a narrow monitoring scope, useful investigation instructions and an explicit review path from insight to pipeline fix.

A pipeline agent is useful when it brings an engineer a better starting point, not merely another notification. Databricks opened Genie ZeroOps in public beta on November 19, making that workflow available beyond its private preview.

The practical question is what happens between finding an issue and accepting a fix. Start with one pipeline where the team understands the failure modes, then evaluate whether the investigation saves work without making review less careful.

What is new in the public beta

Databricks' launch announcement adds shared inboxes, notifications, custom business instructions and one-click PR creation. A workspace administrator enables the beta from the Previews page.

ZeroOps investigates failures and silent data-quality problems rather than waiting for someone to ask a coding assistant what went wrong. The announcement describes gathering evidence, testing hypotheses and proposing fixes. It does not mean every proposed change is correct or should be accepted automatically.

For a first pilot, choose a daily revenue pipeline with a known upstream dependency. Write down the last few incidents, how engineers diagnosed them and what counted as a successful recovery.

Pick the assets and the question

The ZeroOps overview limits monitoring to jobs, pipelines and tables. It does not monitor apps or models, and it does not detect performance or cost regressions. Table data-quality detection requires separately billed data quality monitoring.

That matters for a pilot brief. “Tell us why the revenue table stopped updating” fits the feature. “Reduce our entire Databricks bill” does not.

I would start with a named job and its relevant output table, not every asset the configuring user can see. Record which incidents the pilot should catch and which remain with existing monitoring. Keep the old alert path running until the team has evidence that the new one covers the expected cases.

Measure time to a useful diagnosis, not time to the first notification. A fast message that still requires the engineer to repeat the whole investigation has not solved the original problem.

Check the initial inbox before paging anyone

The setup guide says configuration backfills the previous 24 hours. Notifications support Slack, Teams, PagerDuty and generic webhooks, but not email; an administrator must first create the destination.

Expect the initial inbox to contain historical failures. Reconcile those against incidents the team already handled before forwarding everything to an on-call channel. Otherwise, adoption begins with duplicate work.

For the revenue example, ask the reviewer to classify each insight as already handled, useful new finding or not actionable. Save the reason, especially when a late upstream delivery is expected rather than a defect.

Do not use a quiet inbox as proof that the pipeline is healthy. Compare its coverage with the jobs and tables you intended to include.

Give investigations the missing business context

Custom instructions affect future investigations, not ones already underway. ZeroOps reads the first 20,000 characters, and each report identifies whether it used the instructions.

Keep the guidance specific to a decision an engineer would otherwise have to make. For example, a proposed instruction for the revenue pilot could be:

Check whether the upstream settlement load finished before recommending a retry. Distinguish a late source delivery from a transformation failure. Include the affected reporting date in the diagnosis.

This is more useful than “be accurate and follow best practices.” It names the dependency, the decision and the information the reviewer needs.

When guidance changes, assess a later investigation rather than expecting an existing report to rewrite itself.

Review the fix path, not just the explanation

For workspace code, asking Genie Code to apply a proposed fix can change that code directly. Git-backed fixes follow a different path: the setup guide requires an explicit request and push confirmation, followed by opening and reviewing the pull request.

Make that distinction visible in the team's runbook. For the revenue job, require a reviewer to check the affected date range, expected row counts and treatment of previously processed data before accepting a retry or patch. A plausible diagnosis does not establish that rerunning the job is harmless.

Use a dedicated fix branch and retain the regular review process. Verify the actual destination in the confirmation rather than assuming a button labeled Create PR guarantees every later step has happened.

Share only what the review team needs

The overview notes that shared insights can expose included code excerpts and sample rows to recipients without underlying asset access. Notification copies also sit outside Databricks retention and deletion controls.

Choose recipients deliberately, then confirm that each person understands their role: investigate, review the proposed change or decide the incident is resolved.

Expand the pilot after it produces useful diagnoses, understandable fixes and clear ownership. The best outcome is less time gathering evidence while the team retains responsibility for changing the pipeline.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams explain technical products through clear documentation, practical guides and developer-focused content.

Book a 30-minute call.