Evals are the new tests.
The judgment layer of the .do stack: pinned cases, scored runs, and a regression gate for AI behavior. Its front door, gateway record, and shipped SDK build serve today; its callable run surface is gated; and this deck says which is which, with evidence for both.
For twenty years "safe to ship" had an instrument: the test suite. Then the product grew a model in the middle. Now a prompt tweak changes behavior on inputs nobody tried, a model upgrade improves eight cases and silently ruins three, a retrieval re-index moves answers that were never touched — and the CI gate that guards the rest of the codebase has nothing to say about any of it. The suite still passes. The product changed anyway.
Every serious AI team already knows the answer is evals — the category needs no evangelism. What they have instead is a hand-rolled harness: a script, a spreadsheet, five favorite prompts run by hand before a release. Nobody versions it, nobody pins it, and a verdict from last month is not comparable to one from today, because the cases and scorers drifted with the code. The discipline exists; the infrastructure doesn't.
Three things in this estate are the substrate itself: the studio, the platform, and the runtime. This is deliberately not a fourth. evals.do is the judgment layer the platform composes — filed in the estate's own registry at an exact coordinate: noun Eval, verb evaluate, category ai/evaluation, type service, priority P1. It wears its own door because the builder holding a quality question is a different ICP from the business with an infrastructure decision, and a brand here is one ICP and one motion.
The pack's seam is the object each door holds: functions.do the unit of work, database.do the unit of state, llm.do the unit of inference — this door, the unit of judgment. And the boundary that matters most is drawn by rule, not adjacency: api.qa is the estate's held-out external verifier — a grade of an API surface kept outside the building fleet's write access, because independence is its product. evals.do is the opposite instrument: the first-party suite the team itself authors, versions, and deliberately owns — the fitness function you train and gate against between exams. A serious system wants both; neither substitutes for the other. Registry siblings benchmarks.do (public standardized comparison across the market's models) and experiments.do (variation tracking) hold the adjacent coordinates: a benchmark ranks models in general, an experiment explores variants — an eval grades your system on your cases, and only the last one can gate a ship.
The contract's shape is fixed, and it is the whole product: a case (input, expected, tags), a suite (cases pinned as a versioned dataset), a run (the component, function, workflow, or agent under test, graded against every case by declared scorers), and a verdict (scores against a threshold — comparable to the last run because the suite and scorers are pinned). Pinning is the load-bearing property: an eval you casually edit mid-comparison is a vibe with extra steps. The published SDK already speaks this shape — tests, runs, results, over a local store and a remote API — and the deck states exactly which parts of that loop are posted facts today and which are gated.
The SDK is published and resolvable cold: the npm registry serves the
evals.do package record — versions 0.0.1 and 0.1.0 — described in its
own words as "Evaluation tools for AI components, functions, workflows,
and agents."
The published tarball ships an importable build — checked cold
2026-07-30 and re-verified cold 2026-07-31, and worth stating precisely in a pack where a sibling's did
not: dist/index.js is present at the manifest's declared entry and
carries an EvalsClient with create/run/result methods over a local
store and a remote API defaulting to https://evals.do. One gap stated
plainly: the manifest also declares a types file (dist/index.d.ts)
that is not in the tarball.
The loop this deck sells — submit a suite, get a graded verdict — is
not yet a posted fact. The shipped client targets evals.do as its
remote store, but no call shape is documented at the apex, and this
record does not round a described client up to a served capability. The
claim flips on the first run a cold caller can reproduce from the docs.
Concreteness over adjectives: each door below was checked cold on 2026-07-30 and re-verified cold on 2026-07-31, and carries its own state and its own evidence URL — never one URL evidencing several domains. Serving is a liveness fact, not a tenancy claim: nothing here asserts external tenancy, graded production traffic, or a usage roll. Those publish behind their own gates.
The front door serves. https://evals.do answers 200 with a lit product
lander — "Automate AI Model Evaluation · Continuous Testing for LLMs,"
hero "Continuous AI Evaluation on Every Change," model comparison and
regression-detection positioning, an FAQ, and a Docs CTA into the
platform's documentation hub.
The estate's gateway routes this service as a named record today: GET https://apis.do/evals returns machine-readable JSON — no login — naming
the service evals, its domain evals.do, its category ai, and its
status available.
The lander's Docs CTA target serves: docs.platform.do answers 200. The
URL evidences that door's liveness only — it is the platform's shared
documentation hub, and no dedicated evals doc set is claimed here.
The adjacent execution machinery is published and resolvable cold: the
npm registry serves the ai-evaluate package record — thirteen versions
through 2.4.0 — described in its own words as "Runtime sandboxed
execution of AI-generated code in V8 isolates." That is the execution
half of a code eval: somewhere to run generated code before grading it.
The machine door does not yet serve. The gateway record's own also
link points at evals.do/api — which answers 200 with the human page,
HTML even to Accept: application/json (checked cold 2026-07-30,
re-verified 2026-07-31). A
door whose natural caller is a program does not get to count an HTML
200 as a machine door; the claim flips when the path answers in
structured JSON.
The adjacent execution service is not up. https://evaluate.do
answered HTTP 500 at its apex and 404 at /health on 2026-07-30,
re-verified 2026-07-31 —
while the library it wraps serves its package record cold. The
machinery exists; the door is down; this record says which is which.
Each amber below is the exact distance between what serves and what this door intends to be — named as a checkable gate, not blended into present-tense marketing.
The front door's own headline claims "100+ built-in metrics." No metric catalog serves cold — there is no address where a machine or a skeptic can enumerate them — so this record declines to adopt the figure in its own voice (the estate's rule when a live surface's number outruns what can be verified). The claim flips when the catalog serves; until then the count is the lander's word, not this record's.
The books lag the door: the estate's registry still files evals.do
status: planned while the apex serves. Same class as the database.do
seam, worn the same way — the row is the estate's own record of truth
and gets corrected, not argued with.
No price card is posted and none is implied: evals.do/pricing does
not serve, and the registry row's tier coordinate ("free") is a filing
in the estate's books, not a posted commercial term. Rates bind when
they post at a live surface — never as prose in a deck.
Run counts, suite counts, tenant counts, and the internal-versus- external split are gated. Each figure publishes with its window and base or it does not publish.
Primary motion is B2D: the buyer is a developer who evaluates in the docs and the posted surfaces and converts at the first eval run — no sales motion, no demo call, no procurement. The category is self-explanatory to this ICP — nobody has to be argued into wanting evals in 2026 — so the motion's entire burden is credibility, which is why this deck grades itself in claim states. Secondary is B2A, declared and gated: the natural machine caller is an agent grading its own changes before promoting them, and the first step already serves — the estate's gateway returns this service's record to a machine with no login. The service's own machine door is the amber above, and purchase and settlement for the machine motion gate on the contract surface, exactly as the sibling records state for theirs.
Layer-1 economics at the judgment layer: fixed cost is operating the suite store, run orchestration, and grading pipeline once; each additional tenant, suite, and run rides the same layer at near-zero marginal cost, so blended margin improves with graded volume.
An eval run's cost shape is metered execution plus judge-model calls when scoring is model-graded — and both legs are the pack's own machinery by design: sandboxed execution where code is the subject, the inference gateway where a model is the judge. There is no regulatory floor anywhere in this function: nothing in grading an output reserves a step for a statutory person, so the implementation mix migrates all the way to Code. The commercial frame posts behind the pricing gate above; nothing finer-grained is implied here.
Serving is a liveness fact, not a tenancy claim: each posted URL below evidences that one door resolves — not that anyone occupies it.
evals.do serves — the front door answers 200 with the lit product
lander.
apis.do/evals serves — the estate's gateway lists this service,
status available, to a machine with no login.
The SDK package record resolves cold — published, versioned, and described in its own words.
The importable build ships — dist/index.js present at the declared
entry in the published tarball, verified cold.
The sandbox library's package record resolves cold — the execution half of a code eval, published through thirteen versions.
The cold-reproducible run remains the amber that matters most — the one capability this deck sells is not yet a posted fact, and it flips on the first run a cold caller can reproduce from the docs.
The machine door remains an amber — the path the gateway record points at answers with the human page.
The execution service's own door remains an amber — 500 at the apex, checked cold, while its library serves.
The front door is evals.do — it serves today.
If this was forwarded to you: evals.do is the judgment layer of the startups.studio estate's infrastructure pack — the eval suite as managed infrastructure, sold to the builder who cannot ship a model, prompt, or pipeline change without knowing what it did to quality. It is deliberately not one of the estate's three substrate properties, and it is by rule not the estate's external verifier (that is api.qa, the exam a builder cannot edit; this is the harness a builder owns). What is live is posted with a URL checked cold — the front door, the gateway record, the package record, the shipped build, the sandbox library — and what is not is pending with the gate that flips it, including the seven ambers it wears openly: the cold-reproducible eval run — the one capability this deck sells — the machine door at its own domain, the execution service's down apex, the lander's unverifiable metric count, the registry row that still files the door as planned, the unposted price card, and the gated volumes. Judge it by what is posted, and by how plainly it labels what is not.