SecondChair — self-hosted legal RAG
The constraint that shapes everything
Some documents can't leave the building.
Attorney–client privilege, sealed records, work product. The moment a corpus is legally un-cloudable, the entire managed-AI market disappears — and you build retrieval on hardware you control. That constraint isn't a limitation here; it's the design.
What it is
A self-hosted retrieval-and-analysis pipeline for legal case files. It pairs vector search with a knowledge graph, exposes the corpus to an LLM through a tool server, and runs a multi-pass agent pipeline that doesn't just find passages but produces structured, provenance-checked analysis of a case. Built and shipped in seven days for a state supreme-court appeal, then grown into the multi-pass system described here.
The engagement
The delivery shape is forward-deployed: embed with the customer, learn the workflow from inside it, and ship a production system fast. The seven days weren't a prototype for someone else to harden — the system that left the engagement is the system that runs, and it has supported a matter that reached the Iowa Supreme Court. Working embedded with the practice is what made the privilege model correct: what may surface for a given query is something the practice knows, and an outside specification would get wrong.
Where it generalizes
The instance is legal, but the pattern isn't. Any domain where retrieval must respect an access rule that belongs to the record rather than the query — health, finance, regulated records of any kind — has the same shape: an index over a system of record, a hybrid retrieval layer, and an access constraint enforced where results are assembled. And the delivery model travels further than the architecture: embedding with a customer and shipping a production system in a week applies wherever the gap between the stated requirement and the real need is what sinks projects.
The pipeline
Two stores. Two models. One case, understood.
Documents are parsed, embedded, and graphed, then read by an analysis pass that can search the corpus while it reasons. Retrieval runs entirely on local GPUs.
Vectors and a graph
Hybrid dense + sparse + late-interaction retrieval in Qdrant, beside a Neo4j graph of parties, citations and timeline, queried with Cypher — match passages and traverse relationships.
The right model per pass
Sonnet for high-volume extraction and graph building; Opus reserved for legal reasoning. In-session — no external keys, so a missing model fails loudly.
Retrieval during reasoning
Embedding happens before analysis, so the reasoning pass has live search over the whole corpus while it works — not one prompt's worth of context.
Checked, not trusted
Every finding is validated against retrieved provenance, with automatic retries on failure. It flags a gap rather than invent a citation.
Privilege enforced in retrieval
Queries run through a seven-filter privilege pipeline, and document access is a code-level gate. A result that shouldn't surface is excluded where results are assembled — not hidden in a display step another code path could bypass.
Synced to the source of truth
An integration pulls matters and documents from the practice's case-management system, so the index stays a current reflection of the work the attorney already maintains — not a snapshot taken once.
The stack
BGE-M3 · bge-reranker-v2-m3 GPU-served, local only vLLM · Docker Compose · systemd Where it stands
v2 — the current architecture.
A clean rebuild that turned the 7-day v1 from a search tool into a multi-pass analysis pipeline: three passes, two models, dual store, validated findings.
Three passes, two models
Pass 1 extracts and embeds; Pass 2 builds the graph; Pass 3 runs deep analysis with live retrieval over MCP.
How we got here — what v1 taught us
v1 shipped in seven days and it worked. Running it on a real case exposed the gaps — each became a v2 goal:
| v1 problem (surfaced in real use) | v2 fix |
|---|---|
| Embedding happened after analysis — the reasoner couldn't search while reasoning | Reordered: embed in Pass 1; Opus gets live MCP retrieval in Pass 3 |
| The reranker was configured but never called | Cross-encoder rerank wired into the live path |
| Regex extraction; no key → silent keyword fallback | In-session Sonnet/Opus agents; a missing model fails loudly |
| Vector-only — no relationship reasoning | Neo4j knowledge graph added beside the vectors |
| Shallow analysis; citation scores inflated by duplicate filings | Richer graph schema + dedup |
What comes next
Zero external calls — self-hosted Qwen3
v2's retrieval is already local, but its reasoning still runs on in-session frontier models. v3 swaps them for a self-hosted Qwen3 suite: the last external dependency removed, nothing leaves owned hardware at all. Models pulled and staged; cutover awaiting go-ahead.
staged · gatedEngineered for the input
Real court records are messy. The pipeline expects that.
These aren't clean PDFs. They're scanned filings — e-file stamps banded across every header, exhibits that are photographs, image-only record returns. Off-the-shelf extraction quietly drops half of it. So extraction was built to assume the worst.
A layered fail-loop
Each page runs through a cascade of fallbacks — native text, then OCR, then vision — so a stamped or scanned page is never silently skipped. Every page is classified and routed to the method that can actually read it.
Dialed in to the page
A words-per-page threshold decides when a page needs OCR — tuned on real filings so header-stamped scans stop masquerading as text and slipping through empty.
Images become searchable
Photographs, maps and site plans route to a vision model and join the searchable record — the exhibits plain extraction would leave invisible.
Nothing invisible
A record where every page is readable and every exhibit is findable — the hard part of legal retrieval, handled at ingest instead of discovered at query time.
The query surface
One tool server. Five families of legal question.
The attorney works through a case-work MCP server — a focused set of tools, grouped by the question being asked. The same server feeds the analysis pass.
Search & retrieval
Hybrid semantic + lexical search — by section, party, date, citation, or similarity.
Citation & authority
Authority lookup, citation chains, co-citation clusters, treatment maps across the record.
Argument & strategy
Argument maps, counter-arguments, vulnerability assessment, tracing an issue across filings.
Document structure
Section and sibling-chunk navigation, full-text retrieval, table-of-authorities extraction.
Timeline & procedural
Chronological docket views, filing timelines, and procedural posture — reconstructed from the graph.