An Eight Mile project · open source
ChatAgent
A customer-service agent that decides who should answer before it answers. An LLM triage node routes every conversation to a sales, billing or support specialist, each grounded in your own documents, behind guardrails on the way in and out, and fronted by two tiers of caching.
Runtime
Python 3.12 · FastAPI
Orchestration
LangGraph · LangChain
Retrieval
OpenSearch 2.19.1
Size
~2,400 LOC
researcher_graph
StateGraph
See it think
Watch a conversation get routed.
Three scripted transcripts, replayed exactly as the agent handles them: a handoff to billing with a document lookup, a hostile prompt stopped before it costs anything, and a rephrased question served straight from cache.
POST /chat
thread_id=8m-demo-01
⟩
One question, routed to the right specialist, answered from your own documents.
What it solves
Six ways support chat usually goes wrong.
Every one of these is a design decision in the code, not a setting you have to remember to switch on.
One generic bot answering everything, badly.
Three specialists, one triage brain.
A triage model reads the conversation and returns a typed decision: sales, billing or support with the reason and the context it is handing over. No prose parsing, no guessing.
Confident answers invented on the spot.
Answers grounded in your documents.
Specialists reach for a retrieval tool that searches your own indexed content and returns the passages it used. If retrieval fails it says so instead of improvising.
Anyone can talk your bot out of its job.
A fail-closed security pipeline.
Sixteen prompt-injection patterns, input sanitisation, PII masking and a classifier run before the conversation starts, cheapest checks first, so hostile input costs you nothing.
Personal data ending up in prompts and logs.
PII masked on the way in and out.
Cards, emails, addresses, IBANs, passwords and names are detected by pattern and by zero-shot NER, then replaced with placeholders, before the model call and again after it.
Paying twice for the same question.
Two tiers of response caching.
An in-process exact-match cache in front of a semantic cache over OpenSearch. Rephrased repeats are served from vectors in milliseconds, scoped to the conversation that asked.
A bot that forgets you mid-sentence.
Conversations that persist.
Every conversation is a thread with a durable checkpoint. Reconnect an hour later and the specialist still has the context, and the cache still knows it is yours, not someone else’s.
By the numbers
defaults, all configurable
3
Specialists
sales · billing · support
2
Cache tiers
exact match, then semantic
16
Injection patterns
checked before any model call
10
PII labels
pattern + zero-shot detection
k=4
Chunks retrieved
cited back as sources
1
Thread = 1 memory
never crossed between users
What you get
Built like infrastructure, not a demo.
The parts that matter in production — typed contracts, validation on both sides, durable conversations and cost control — are in the repository, not on a roadmap.
Typed routing
Handoffs are a Pydantic decision object, not a sentence the next model has to interpret.
Two-sided validation
Input is checked before the graph runs; every response is scanned for harmful content and leaked PII before it reaches the customer.
Thread-scoped memory
thread_id is the unit of identity everywhere — graph state, checkpoints and cache partitions.
Runs on your infrastructure
A single OpenSearch node via docker compose. Your documents and conversations stay in your stack.
Quality kept honest
An LLM judge scores answers on correctness, relevance, clarity and completeness against a fixed dataset to catch drift.
Cost controls
Token budgeting with tiktoken, a cheaper fallback model, rate limiting, and cache hits that cost nothing at all.
Layout
Three packages, dependencies flowing one way.
api → app → core. No DI framework: everything is constructed once in the FastAPI lifespan and passed down by constructor inside the domain.
src/api
·
FastAPI app + lifespan
·
chat · health · metrics · cache
·
Pydantic request/response models
depends on src/app
src/app
·
agents/researcher: graph, nodes, routes, tools
·
security: sanitiser, pii, guard, validator
·
common: cache, rag, observability, evaluation
depends on src/core
src/core
·
config: pydantic-settings
·
logging: JSON formatter
·
models + store/vectorstore
The agent graph
One StateGraph, three conditional edges.
Triage decides the destination with with_structured_output, so routing is a typed Pydantic decision rather than parsed prose. Specialists loop through a ToolNode until they stop asking for tools.
Active route
Conditional edge
Terminates
Control flow
START → triage → {sales | billing | support | END}
specialist → tools (if tool_calls) → back to the calling specialist
specialist → END (if no tool_calls)route_from_triage
Reads current_agent, falls through to end.
should_continue
Is the last message an AIMessage with tool_calls?
route_from_tools
Returns to whichever specialist called the tool.
Read the source
Ten files that explain the whole thing.
Excerpts straight from the repository: the graph, the routing, the state contract, the security pipeline and both cache tiers.
chatagent / src
src/app/agents/researcher
src/app/security
src/app/common/cache
src/app/agents/researcher/agent.py
graph = StateGraph(ResearcherState) graph.add_node('triage', nodes.triage_agent) graph.add_node('sales', nodes.sales_agent) graph.add_node('billing', nodes.billing_agent) graph.add_node('support', nodes.support_agent) graph.add_node('tools', nodes.tool_node) graph.add_edge(START, 'triage') graph.add_conditional_edges( 'triage', routes.route_from_triage, {'sales': 'sales', 'billing': 'billing', 'support': 'support', 'end': END}, ) graph.add_conditional_edges( 'billing', routes.should_continue, {'tools': 'tools', 'end': END} ) # sales and support wire up identically graph.add_conditional_edges( 'tools', routes.route_from_tools, {'sales': 'sales', 'billing': 'billing', 'support': 'support'}, ) return graph.compile(checkpointer=saver)
The whole topology in one place. Nodes are plain methods; every branch is a named function, so routing is testable without a model.
src/app/agents/researcher/routes.py
class ResearcherRoutes: def route_from_triage( self, state: ResearcherState ) -> Literal['billing', 'sales', 'support', 'end']: agent = state['current_agent'] routable_agents: list[ResearcherAgent] = ['billing', 'sales', 'support'] if agent in routable_agents: return agent return 'end' def route_from_tools( self, state: ResearcherState ) -> Literal['sales', 'billing', 'support']: """Route back to whichever specialist invoked the tool call.""" return cast(Literal['sales', 'billing', 'support'], state['current_agent']) def should_continue(self, state: ResearcherState) -> Literal['tools', 'end']: """Check if should continue to tools or end.""" last_message = state['messages'][-1] if isinstance(last_message, AIMessage) and last_message.tool_calls: return 'tools' return 'end'
Three conditional edges, no cleverness. route_from_triage falls through to end; should_continue only asks whether the last message wanted a tool.
src/app/agents/researcher/state.py
class ResearcherState(TypedDict): """State passed between nodes in the researcher graph.""" messages: Annotated[list[BaseMessage], add_messages] current_agent: ResearcherAgent | None handoff_reason: str context_summary: str error: str | None retry_count: int model_used: str class TriageUpdate(TypedDict): current_agent: ResearcherAgent handoff_reason: str context_summary: str messages: list[BaseMessage] class SpecialistUpdate(TypedDict): messages: Annotated[list[BaseMessage], add_messages] current_agent: ResearcherAgent
The contract between nodes. messages carries the add_messages reducer so nodes return the delta and LangGraph does the appending.
src/app/agents/researcher/schemas.py
type ResearcherAgent = Literal['sales', 'billing', 'support', 'end'] class HandoffDecision(BaseModel): handoff_to: ResearcherAgent = Field(description='Which agent to hand off to') reason: str = Field(description='Reason for handoff') context: str = Field(description='Key context to pass to the next agent')
Routing is a value, not a sentence. with_structured_output binds this model to the triage call, so an invalid destination cannot be expressed.
src/app/agents/researcher/nodes.py
def triage_agent(self, state: ResearcherState) -> TriageUpdate: """Initial triage to route the customer query to.""" messages: list[BaseMessage] = [ SystemMessage(content=TRIAGE_AGENT_PROMPT), *state['messages'], ] decision = cast( HandoffDecision, self.triage_llm.invoke(messages), ) if decision.handoff_to == 'end': ... return { 'context_summary': decision.context, 'handoff_reason': decision.reason, 'current_agent': decision.handoff_to, 'messages': [ AIMessage( f'[ TRIAGE ] Transferring to {decision.handoff_to}. {decision.reason}' ) ], }
The triage node. The decision comes back typed, so the branch below it is an equality check rather than string sniffing.
src/app/agents/researcher/tools.py
class ResearcherTools: def __init__(self, rag: Rag) -> None: self.rag = rag self.get_relevant_documents = tool(self._get_relevant_documents) def _get_relevant_documents(self, query: str) -> str: """Search for information relevant to the query.""" return self.rag.ask(query)
One tool, bound with bind_tools and run by a prebuilt ToolNode. The docstring is the tool description the model actually reads.
src/app/security/security_pipeline.py
@traceable(name='security_check_input') def check_input(self, input: str) -> InputCheckResult: """Process the user input prompt through security checks.""" notes: list[str] = [] # 1. Check for suspicious input suspense = self.input_sanitiser.is_suspicious(input) if suspense.is_suspicious: notes.append(suspense.reason) if suspense.reason is not None else None return InputCheckResult(is_allowed=False, security_notes=notes, cleaned_text='') # 2. Sanitise the input cleaned = self.input_sanitiser.sanitise(input) # 3. Mask PII cleaned_masked = self.pii_detector.mask(cleaned) # 4. Go through security guard security_check_result = self.security_guard.security_check(cleaned_masked) if not security_check_result.safe: notes.append(security_check_result.reason) return InputCheckResult(is_allowed=False, security_notes=notes, cleaned_text='') # 5. Security checked return InputCheckResult( is_allowed=True, security_notes=notes, cleaned_text=cleaned_masked )
Fail-closed and deliberately ordered: two free regex passes, then local PII masking, and only then the classifier that costs a model call.
src/app/security/input_sanitiser.py
class InputSanitiser: """Sanitise user input before processing.""" INJECTION_PATTERNS = [ re.compile( r'ignore\s+(all\s+)?(previous|prior|above)\s+instructions', re.IGNORECASE ), re.compile(r'you\s+are\s+now\s+(a|an|in)\s', re.IGNORECASE), re.compile(r'reveal\s+(your|the)\s+(system\s+)?(prompt|instructions)', re.IGNORECASE), re.compile(r'</?(system|assistant|user)>', re.IGNORECASE), re.compile(r'\bDAN\s+mode\b', re.IGNORECASE), # …16 in total ] def is_suspicious(self, text: str) -> IsSuspiciousResult: """Check if input contains suspicious patterns""" for pattern in self.INJECTION_PATTERNS: if pattern.search(text): return IsSuspiciousResult( is_suspicious=True, reason=f'Suspicious pattern detected: {pattern.pattern}', ) return IsSuspiciousResult(is_suspicious=False, reason=None)
Compiled once at class scope. Rejection names the pattern that matched, so a false positive is a one-line fix rather than an investigation.
src/app/common/cache/hash_cache.py
async def get(self, query: str, thread_id: str) -> str | None: """Get cached response if the same query exists and has not expired""" query_hash = self._hash_query(query) with self._lock: entry = self.cache.get(query_hash) if entry is None: return None if entry['thread_id'] != thread_id: return None if self._is_expired(entry, int(time()), self.ttl_seconds): del self.cache[query_hash] return None entry['hits'] += 1 self.cache.move_to_end(query_hash) return entry['response']
L1. get() mutates, LRU reorder and hit count, so it is not safe to run concurrently from the threadpool that serves sync endpoints. Hence the lock.
src/app/common/cache/semantic_cache.py
async def get(self, query: str, thread_id: str) -> str | None: """Find the one most semantically similar query WHERE thread_id equals the current thread AND timestamp is newer than the expiry cutoff""" expires_after = int(time()) - self.ttl_seconds results = await self.vectorstore.asimilarity_search_with_relevance_scores( query=query, k=1, efficient_filter={ 'bool': { 'filter': [ {'term': {'metadata.thread_id': thread_id}}, {'range': {'metadata.timestamp': {'gte': expires_after}}}, ] } }, ) if not results: return None doc, score = results[0] # Raw OpenSearch kNN score, not a cosine similarity: the score formula # depends on the index engine and space_type, so comparing it directly # avoids hardcoding a conversion that silently breaks if either changes. # Monotonic in cosine either way, so it is still a valid cutoff. if score < self.score_threshold: return None
L2. The filter is the whole point: nearest neighbour, but only inside this thread and only inside the TTL window. Note the comment on the threshold.
Security pipeline
Ordered so hostile input costs nothing.
check_input is fail-closed and deliberately sequenced: two free regex passes, then local PII masking, and only then the classifier that spends a model call. check_output runs the mirror image.
01
is_suspicious
Sixteen compiled prompt-injection regexes: instruction overrides, role reassignment, system-prompt extraction, DAN markers, fake <system> tags. A match rejects the request and names the pattern.
check_input
02
sanitise
Strips --- and === runs and defuses {{ }} template braces so nothing downstream treats user text as structure.
check_input
03
mask
Presidio pattern recognisers plus a GLiNER zero-shot recogniser over ten labels, anonymised through Presidio’s AnonymizerEngine.
check_input
04
security_check
An LLM classifier returns a JSON safe/reason verdict. Last, because it is the only step that costs a model call.
check_input
05
validate
Nine harmful-content regexes. A match replaces the whole response with [CONTENT BLOCKED] rather than editing around it.
check_output
06
second pii scan
The response is masked again on the way out, in case the model echoed something the input scan let through.
check_output
Both entry points are LangSmith-traced. This is the best-tested area of the repository: unit, integration and regression suites across four modules, with the regression suite replaying a fixed dataset against live models to catch quality drift.
Caching
An L1/L2 pair behind one abstract base.
Cache declares get, set, get_stats and purge_expired. Every entry is keyed by (query, thread_id) so one conversation can never be served another’s response.
L1
HashCache
in-process
Key
md5(casefolded query)
OrderedDict LRU, bounded by TTL and max_entries
TTL checked lazily on read; LRU eviction on write
A threading.Lock guards the read-modify-write in get()
L2
SemanticCache
OpenSearch kNN
Key
sha256(thread_id + normalised query)
Filtered k-NN at k=1 with an efficient_filter on thread_id and a TTL cutoff
Hits accumulate in a bounded deque and flush as one bulk update
One maintenance task flushes hits → purges expired → evicts, in that order
Caveat carried in the code
The threshold compares a raw OpenSearch kNN score, not a cosine similarity. The score formula depends on the index engine and space_type, so it needs recalibrating if either changes. It is monotonic in cosine either way, which is why it is still a valid cutoff.
Storage & retrieval
Two indices, pinned by hand.
Both are created from explicit mappings with dynamic: strict rather than letting langchain infer them — the vector method and every metadata field type is declared instead of inherited from whatever the first document happened to contain.
chatagent_documents
hnsw · engine lucene · space_type cosinesimil · 1536 dims. Metadata: source, document_id, page, chunk_index, title.
Rag chunks with RecursiveCharacterTextSplitter at 1000/200 on paragraph-first separators, stamps source and indexed_at, retrieves at k=4 and formats results as [Source N: origin] blocks. Retrieval failures degrade to a plain sentence rather than raising into the graph.
chatagent_cache
query · response (stored, not indexed) · thread_id (keyword) · timestamp (long) · hits (integer). Document IDs are sha256(thread_id + normalised query), so an exact repeat overwrites rather than duplicates.
Reads and writes are async; admin calls use the sync client from a worker thread. One background maintenance task flushes hit counts, purges expired entries, then evicts below the cap, in that order, because eviction ranks on hit counts.
Run it
Three commands, then a scratch driver.
chatagent
$
docker compose up -d opensearch
$
uv sync
$
uv run uvicorn api.main:app --app-dir src --reload
uv run python main.py # seeds the index and pushes five messages through the graph
The whole thing in one pass
main.py at the repository root is the fastest way to watch it work without an HTTP client in front of it: it seeds two sample documents into the RAG index, opens a SqliteSaver, renders researcher_graph.png and pushes five messages through the graph on one thread_id, exercising triage, handoff, tool use and conversational memory in a single run.
Read main.py on GitHub →Conventions
Non-standard on purpose, enforced across the tree.
Formatter
ruff · 2-space indent · 88 columns · single quotes
Spelling
British in identifiers: sanitise, normalised, cost_optimisation
Results
Frozen dataclasses and TypedDicts, never tuples or bare dicts
Wiring
Constructor injection in the domain; module-level state only in api/main.py
Typing
Strict, with targeted pyright: ignore[...] on untyped langchain surfaces
Tests
unmarked = unit · -m integration = real network · -m regression = quality drift
Current state
What is finished, and what is honestly not.
The HTTP layer has landed: routers are registered, the chat endpoint runs the full security, cache, agent and metrics path, and the primary/fallback model chain is real. Nothing blocking is left. What remains is single-process scaling, helpers that were built and never attached, and config drift.
The rate limiter is per-process
slowapi is constructed with a key function and no storage_uri, so every uvicorn worker keeps its own counter and N workers means an N× looser limit. Redis-backed storage is the first requirement to run more than one process.
Metrics and the L1 cache do not span workers
The semantic cache moved to OpenSearch and is shared, but HashCache is an in-process OrderedDict and MetricsCollector a plain dict, so /metrics reports one worker’s view. Semantic hit counts batch through a deque(maxlen=1000) that silently drops overflow.
The resilience helpers are still on the shelf
Model fallback is live — Models applies bind_tools and with_structured_output underneath with_fallbacks across gpt-4o-mini → claude-sonnet-4-5 → gpt-4o. CircuitBreaker, with_retry and with_fallback_llm are written and unit tested but attached to no call path.
fallback_chain.py is superseded and still buggy
Models' fallback chain replaced it, yet FallBackChain remains in the tree with its literal-key bug intact: a cache hit returns self.cache['query'], the string rather than the variable. Nothing but its own tests imports it.
Postgres is a dependency, not a code path
langgraph-checkpoint-postgres and psycopg are installed, but nothing imports PostgresSaver — SqliteSaver against checkpoints.db is what actually runs. Wire it or drop the dependencies.
The example env file has drifted from Settings
.env.example still misspells MAX_RETIRES, so max_retries never binds and stays 0; still lists DATABASE_URL, which Settings no longer reads; and names the tracing keys LANGSMITH_* while Settings expects LANGCHAIN_*. The README intro also still advertises circuit-breaking.
Behind the project
Built by Eight Mile in London, as part of our software development work — the same engineers who build and run systems like it for clients.