AI Products
EvalKit
A product for catching LLM regressions before they become user problems.
2026 / AI Products / FastAPI / SQLAlchemy / PostgreSQL / Neon / Stripe / Railway
Quick read
A clear breakdown of the product, the reasoning behind it, and the implementation choices that matter.
- Year
- 2026
- Category
- AI Products
- Status
- Active prototype
Overview
EvalKit is a multi-tenant SaaS for regression testing LLM behavior. It gives teams a deterministic engine built around suites, cases, checks, and runs so they can catch schema drift, forbidden content, length regressions, and silent capability loss before those changes reach users.
Why it exists
LLM features drift in ways that are easy to miss in a quick review. Most teams do not have a repeatable way to measure whether the output still meets the product contract over time.
How it works
EvalKit turns evaluation into a workflow teams can actually live with: define suites and cases, attach checks, run them, and review regressions before they ship.
Built with
FastAPI / SQLAlchemy / PostgreSQL / Neon / Stripe / Railway
What I focused on
- Built a deterministic check engine for repeatable regression tests.
- Designed the SaaS for multi-tenant projects and scoped access.
- Handled billing, metering, and webhook-driven subscription state.
- Shipped a Railway deployment path without Docker to keep iteration fast.