Eight Mile · AI integration · LLMOps
The half of AI that keeps it running.
A prototype that answers well in a demo is the easy half. LLMOps is the rest: caching so a repeat question costs nothing, checks on what goes into a model and what comes out, evaluations that prove a change made things better, a fallback when a provider has a bad day, and a budget the feature cannot outspend.
What we build
LLMOps, built as software.
Two-tier caching
An exact lookup on a hash of the request, then a semantic lookup on its embedding. The cheap check runs first, and a cache outage falls through to the model rather than failing.
Guardrails in and out
Inputs checked for injection and abuse with free pattern checks before any model-based classifier is paid for; outputs checked before they reach a user.
PII masking
Names, emails, phone numbers and account details masked locally before a prompt leaves your network.
Model fallback
A chain of models behind one interface: when the first times out or errors, the request moves to the next, and the switch is recorded.
How it works
The same steps, in the order they run.
01
Measure
Every call is logged with its model, tokens, latency, cost and cache result, so there is a baseline before anything is tuned.
02
Evaluate
A set of real inputs is scored — by exact rules where the answer is checkable, by a model acting as judge where it is not — and the score is the gate.
03
Change
A new prompt, model or retrieval setting is tried against that set, and ships only if the score holds and the cost does not climb.
04
Watch
Cost per workspace, cache hits, fallbacks and guardrail refusals are tracked in production, so a drift shows up in a chart rather than a complaint.
Controls
What makes it safe to ship.
Decided before the first prompt is written, and tested like the rest of the code.
Budgets that refuse
A monthly spend limit per workspace or customer, counted from a usage ledger. When it is reached, calls stop and the product says so, rather than the bill arriving later.
Metering you can bill from
Every successful call records who made it and what it cost, so AI usage can be passed on to your own customers or capped per plan.
Closed on safety, open on cache
A guardrail that cannot run blocks the request; a cache that cannot be reached is skipped. Each failure mode is chosen, not inherited.
Evaluations in the repository
The test set and the judge’s rubric live beside the code and run in the pipeline, so nobody can change a prompt without seeing the score.
Proof
Where you can see it working.
Our own products, each with a page of its own that shows how it is built.
Straight answers
The questions people ask about it.
The rest of our AI work
Next step
Tell us about the AI feature you already have.
What it does, which model it calls and what worries you about it — the cost, the answers or the next change. We will tell you what we would measure first.