Product Strategy
EvalKit Build Thread / Part 1 of 3
Introducing EvalKit
EvalKit started from a simple frustration: once an LLM feature ships, getting another response from the model is easy. Knowing whether the system still behaves the way the product needs it to is the hard part. This post lays out the product model I wanted to build, the evaluation objects that make it usable, and what I think matters most next.
Reading note
A clearer write-up of the product thinking, system choices, and tradeoffs behind the build.
In this post
If you cannot tell whether the model improved, you are shipping guesses with a nicer interface.
Outline
- The product problem EvalKit is solving.
- How suites, cases, checks, and runs fit together.
- Who the product is for and where it fits.
- What is currently shipped.
- What comes next and how the series connects.
The Problem
LLM teams usually discover regressions late, because outputs are easy to glance at and hard to measure. A prompt change can quietly alter JSON shape, length, tone, safety boundaries, or the exact capability that made the feature useful in the first place. Spot checks catch the obvious breakage, but they don't leave you a repeatable record of what changed.
EvalKit exists to make that record explicit. It's not trying to test every possible response — it's a harness that makes output drift visible before it turns into a product problem.
How EvalKit Works
Suite
|
v
Cases
|
v
Checks
|
v
Run
|
v
Regression ReportA suite groups a product flow. A case captures one concrete prompt or interaction. A check describes what good looks like for that case: valid schema, forbidden content, maximum length, or a capability constraint the output should preserve. A run executes the suite, records results, and shows where the output drifted.
- Suites keep the scope aligned with a real product surface.
- Cases make the test data explicit and reusable.
- Checks let different failure modes stay separate instead of blending into one vague score.
- Runs preserve history so regressions can be compared over time.
Who It Is For
EvalKit is aimed at teams shipping LLM features inside SaaS products, internal tools, support automation, and agent workflows — anywhere the model is part of a larger system and the team needs confidence that a change in one place didn't break behavior somewhere else.
That's product teams who need a simple evaluation ritual, engineers who want regression coverage they can reason about, and founders who need to prove the AI layer is doing real work instead of just producing plausible-sounding text.
Current Status
The current build establishes the core evaluation objects, the SaaS shape, and the deterministic flow through the engine. The immediate goal was getting the product model right enough that future features can attach to it without rewriting the foundation.
The launch post is part of the work too — it sets the vocabulary for the rest of the series and gives the other two posts a stable reference point: the evaluation argument and the engine implementation.
Roadmap And Cross-Links
Those three posts are meant to read as one sequence. The launch post explains the product, the second post argues for the need, and the third post documents the implementation tradeoffs that made the product more solid.