An Eight Mile project · open source

ChatAgent

A customer-service agent that decides who should answer before it answers. An LLM triage node routes every conversation to a sales, billing or support specialist, each grounded in your own documents, behind guardrails on the way in and out, and fronted by two tiers of caching.

Runtime

Python 3.12 · FastAPI

Orchestration

LangGraph · LangChain

Retrieval

OpenSearch 2.19.1

Size

~2,400 LOC

researcher_graph

StateGraph

ENDENDENDENDSTARTtriagestructured decisionsalesrag + toolsbillingrag + toolssupportrag + toolstoolsk=4

See it think

Watch a conversation get routed.

Three scripted transcripts, replayed exactly as the agent handles them: a handoff to billing with a document lookup, a hostile prompt stopped before it costs anything, and a rephrased question served straight from cache.

POST /chat

thread_id=8m-demo-01

 

One question, routed to the right specialist, answered from your own documents.

What it solves

Six ways support chat usually goes wrong.

Every one of these is a design decision in the code, not a setting you have to remember to switch on.

One generic bot answering everything, badly.

Three specialists, one triage brain.

A triage model reads the conversation and returns a typed decision: sales, billing or support with the reason and the context it is handing over. No prose parsing, no guessing.

Confident answers invented on the spot.

Answers grounded in your documents.

Specialists reach for a retrieval tool that searches your own indexed content and returns the passages it used. If retrieval fails it says so instead of improvising.

Anyone can talk your bot out of its job.

A fail-closed security pipeline.

Sixteen prompt-injection patterns, input sanitisation, PII masking and a classifier run before the conversation starts, cheapest checks first, so hostile input costs you nothing.

Personal data ending up in prompts and logs.

PII masked on the way in and out.

Cards, emails, addresses, IBANs, passwords and names are detected by pattern and by zero-shot NER, then replaced with placeholders, before the model call and again after it.

Paying twice for the same question.

Two tiers of response caching.

An in-process exact-match cache in front of a semantic cache over OpenSearch. Rephrased repeats are served from vectors in milliseconds, scoped to the conversation that asked.

A bot that forgets you mid-sentence.

Conversations that persist.

Every conversation is a thread with a durable checkpoint. Reconnect an hour later and the specialist still has the context, and the cache still knows it is yours, not someone else’s.

By the numbers

defaults, all configurable

3

Specialists

sales · billing · support

2

Cache tiers

exact match, then semantic

16

Injection patterns

checked before any model call

10

PII labels

pattern + zero-shot detection

k=4

Chunks retrieved

cited back as sources

1

Thread = 1 memory

never crossed between users

What you get

Built like infrastructure, not a demo.

The parts that matter in production — typed contracts, validation on both sides, durable conversations and cost control — are in the repository, not on a roadmap.

Typed routing

Handoffs are a Pydantic decision object, not a sentence the next model has to interpret.

Two-sided validation

Input is checked before the graph runs; every response is scanned for harmful content and leaked PII before it reaches the customer.

Thread-scoped memory

thread_id is the unit of identity everywhere — graph state, checkpoints and cache partitions.

Runs on your infrastructure

A single OpenSearch node via docker compose. Your documents and conversations stay in your stack.

Quality kept honest

An LLM judge scores answers on correctness, relevance, clarity and completeness against a fixed dataset to catch drift.

Cost controls

Token budgeting with tiktoken, a cheaper fallback model, rate limiting, and cache hits that cost nothing at all.

Where your data lives

Your documents, your conversations, your infrastructure.

Retrieval and caching run on an OpenSearch node you host. Conversations checkpoint locally. Personal data is masked before the model call, and the response is scanned again before it reaches the customer.

Self-hosted vector store and cache

PII masked before the model sees it

Cache partitioned per conversation

Every run traceable in LangSmith

Behind the project

Built by Eight Mile in London, as part of our software development work — the same engineers who build and run systems like it for clients.