How a Boise agri-tech team put real evals around its crop-recommendation model
An agronomy platform's nitrogen recommendations are the product, and updates shipped on spot checks and gut feel. After a bad regional miss, the team built an eval suite and an experiment log inside the workspace. Every model change now runs 340 regression cases first, and two bad releases were caught before a grower saw them.
The situation
The product is the model. When the nitrogen recommendation is right, growers save real money; when it drifted wrong for one soil region two seasons ago, the company spent a winter rebuilding trust with every dealer in that territory. The release process that let it happen was honest but thin: the ML lead shipped an update, an agronomist spot-checked a dozen fields, and everyone hoped.
The founder-CTO's diagnosis was blunt: an ML-driven product without evals is a product without QA. But building an eval harness from scratch competed with roadmap work two PMs could not spare, so it stayed a to-do for a year.
The turn
The eval suite got built inside the workspace in three weeks instead of the quarter it had been scoped at. Agronomist-validated historical cases, 340 of them across soil types, weather years, and crop rotations, became a regression suite every candidate model runs before release. Turnaround is six hours, which means the ML lead runs it on every serious experiment, not just release candidates.
The quieter change was the experiment log. Every model run, threshold change, and outcome records itself with rationale, so the team's institutional memory of what was tried and why stopped living in the ML lead's head. Two releases have been stopped by the suite since: one nitrogen regression on sandy soils, one seeding-rate error in a dry-year scenario. Both would have reached growers under spot checks.
“We are an ML company that shipped on vibes and a dozen spot checks. The eval suite is the first time I can say what a model change does before a grower finds out for us. Two catches paid for the whole thing.”
How Second Axis fits
Evals as a workflow, not a project
Historical agronomist-validated cases became a 340-case regression suite built in weeks inside the workspace, not a quarter-long side project.
An experiment log that fills itself
Every model run and threshold change records its rationale and outcome, building a queryable history of what was tried and why.
Six-hour eval turnaround
Fast enough that evals run on every serious experiment, which is the difference between a gate and a culture.
PM-legible model changes
Eval results arrive as plain-language deltas, so the two PMs debate grower impact instead of decoding metrics.
Results
| Area | Before | After |
|---|---|---|
| Regression coverage per model release | ~12 spot checks | 340 cases |
| Bad releases reaching growers | 1 major in 2 seasons | 0, two caught pre-release |
| Model experiments logged with rationale | Ad hoc | 100% |
| Eval turnaround on a candidate model | Days, manual | 6 hrs |
| Time to build the eval harness | Scoped at a quarter | 3 weeks |
What came next
The territory that took a winter to win back had a clean season, and the dealer conversations there changed accordingly. The founder-CTO's summary to the board was one line: the model now has QA, and the proof is the two releases growers never saw.
Company details anonymized at the customer's request.