docs/setup.md

Setup Guide

Scope

This guide covers installation and configuration of PrecedentAI's legal research and drafting workflows:

1. LangGraph graph skeleton (routing, checkpointing, human-in-the-loop)
2. Legal retrieval — keyword floor and ChromaDB + LaBSE at corpus scale
3. Judgment summarization — structured, plain-language, and Hindi bilingual variants
4. Safe drafting — interrupt-gated, citation-checked
5. Gated LLM stage — facts pass, translation, drafting, routing, planning (Groq)
6. Agentic patterns — self-corrective retrieval, multi-agent supervisor
7. Serving layer and chat UI — FastAPI, SSE streaming, usage ledger

Transcription is intentionally outside the current scope.

Prerequisites

• Python 3.12 or newer
• uv package manager
• A local editor and terminal
• GPU is optional: LaBSE inference and index builds run on CPU, just slower

Install Dependencies

From the repository root:

uv sync

The dependency metadata is in pyproject.toml.

Environment

Copy the example environment file if you want the LLM stage:

cp .env.example .env

Nothing requires an API key. Retrieval, summarization, drafting, and the whole test suite run on the deterministic floor with no keys and no network. The selected provider's key is used only when LEGAL_AI_ENABLE_LLM=true:

• GROQ_API_KEY — the default provider
• OPENAI_API_KEY — when LEGAL_AI_LLM_PROVIDER=openai
• NVIDIA_API_KEY — when LEGAL_AI_LLM_PROVIDER=nvidia
• AR_API_KEY — when LEGAL_AI_LLM_PROVIDER=agentrouter

Do not commit .env or any real API keys.

Data Setup

Public Adalat AI Datasets

The Text-track retrieval experiments can start with public datasets from the Adalat AI Hugging Face organization:

• adalat-ai/Indian-Legal-Retrieval-Generation
• adalat-ai/in22-legal
• adalat-ai/indian-legal-exam-benchmark

Use only data whose license and usage terms permit the intended experiment.

Local Legal Documents

When adding local material, organize it by source and language:

data/legal/
├── constitution/
├── ipc/
├── judgments/
│   ├── hi/
│   ├── ta/
│   ├── te/
│   ├── kn/
│   ├── mr/
│   └── bn/
└── metadata/

Every document should have a stable source ID, court, year, language, and license/provenance metadata.

Audio Data

No audio setup is required for the current Text-track phases. Add data/speech/ only when starting the separate transcription work.

Run the Graph

With no index and no keys, the graph runs on the offline fixture floor:

export LEGAL_AI_RETRIEVAL_BACKEND=keyword
uv run python -m agent.core      # routes three sample requests and prints state

With both indexes built (next section) the default backend is fused and the same command searches the real corpus. The API serves both:

uv run --extra serve --extra ml --extra llm uvicorn api.main:app
# GET /health reports the live backend and whether the index is on disk

Retrieval Backends

Three backends sit behind one contract (retrieval_backend config knob):

• fused (default) — independent ChromaDB semantic search, SQLite FTS5/BM25 search and exact identity lookup, merged at judgment level with supporting passages.
• semantic — ChromaDB + LaBSE over the indexed corpus, for dense-only comparisons.
• keyword — a 3-document in-memory fixture with zero dependencies. The offline floor the test suite runs on (tests/conftest.py pins it) and the escape hatch when there is no index on disk. It is a fixture, not a search engine: it scores 0 against real questions.

Build the retrieval indexes — it is a separate, explicit step so model downloads never happen at graph-compile or test time:

uv sync --extra ml                                # one-time, heavy (torch)
uv run --extra ml python scripts/build_index.py   # embeds the corpus (config.corpus_path)

For an existing vector index, add the independent lexical index without embedding the corpus again:

uv run python scripts/build_index.py --lexical-only

The SQLite search sidecar lives alongside the Chroma collection. It is separate from the parent-text reassembly sidecar below. Build it again when the corpus changes; the fused backend reports a missing sidecar rather than pretending to perform corpus-wide keyword search.

--corpus defaults to LEGAL_AI_CORPUS_PATH. Pass --corpus data/legal/seed/judgments.jsonl for a 7-document smoke index. Indexes are gitignored; budget disk and build time for the full collection. The source-aware profile below stores text and rich provenance once in SQLite, with vectors and filter metadata in Chroma. Builds stream bounded batches and resume committed progress. Rebuild in a separate directory when changing the corpus or embedding model; the builder rejects incompatible reuse.

Full-judgment reassembly (summarize, citation checks) reads parent texts through a SQLite sidecar next to the corpus JSONL, created on first use.

Summarization (Phase 3)

Summarization runs automatically as a retrieve -> summarize chain: a "Summarize the holding in..." request retrieves the judgment first, then the node reassembles the full parent text from the chunked corpus (corpus_path config knob) and extracts structured sections from its SCR headnote. Every returned quote is a verbatim source span carrying its character offset.

Score the summarizer against the hand-labeled landmark references:

uv run python -m eval.run_summarize_eval

The golden set lives in data/eval/summarization.jsonl (5 judgments: case_id, reference sections, key_points). Targets: quote support >= 0.90, holding ROUGE-1 >= 0.50, key-point coverage >= 0.60, section F1 >= 0.40.

Output variants

The summary output is a config knob (LEGAL_AI_SUMMARY_VARIANT, default structured):

LEGAL_AI_SUMMARY_VARIANT=plain   # glossary-simplified, short sentences
LEGAL_AI_SUMMARY_VARIANT=hindi   # bilingual: Hindi labels + verified terms

Score each variant against the golden references:

uv run python -m eval.run_summarize_eval --variant plain
uv run python -m eval.run_summarize_eval --variant hindi

Every runner also prints its LLM spend footer (deterministic runs report none). Add --llm to the summarize runner to run the anchor-gated LLM facts pass per record and score facts_llm_coverage (token-level recall over the golden reference facts — informational, not a gated target, since valid narratives legitimately pick different aspects to cover):

uv run python -m eval.run_summarize_eval --llm

If Groq's free-tier daily token budget is exhausted, the --llm pass degrades to the deterministic floor (llm_ok=0.000) and the spend footer reports the failed calls — rerun after the window resets.

Testing

uv run --all-extras pytest -q                 # small offline behavioral suite
LEGAL_AI_RUN_EMBEDDING_TESTS=1 uv run --all-extras pytest tests/test_semantic.py -q

The default suite uses small fixtures and fake model clients. It covers complete chat and streaming turns, isolated conversation history, approval and rejection, source reading, semantic-index round trips, cited answer fallbacks, summary/OCR regressions, translation gates, provider retries, supervisor delegation, and all three evaluation harnesses. Two real-LaBSE checks are opt-in. Install all extras for full offline coverage; otherwise API/index tests may be skipped.

Keep a test when it protects a user-visible behavior, a distinct failure mode, or a bug found in practice. Extend an existing workflow test when it can check another part of the same result. Import smokes, framework/schema checks, visual graph topology, repeated helper permutations and test-count targets do not belong in this suite. Model quality is measured by the evaluation harnesses.

Retrieval Evaluation

uv run --extra ml python -m eval.run_eval --backend fused      # the product default
uv run python -m eval.run_eval --backend keyword               # fixture floor
uv run --extra ml python -m eval.run_eval --backend both      # dense versus hybrid

The runner exits nonzero when any docs/benchmark.md target is missed (CI-friendly). Two golden sets live in data/eval/:

• golden_retrieval_scaled.jsonl — 8 queries originally labelled against the legacy corpus by case_id; the unchanged labels also evaluate the rebuilt snapshot (the runner's default)
• golden_retrieval.jsonl — 10 queries over the 7-doc seed corpus, kept for provenance

Each line is one JSON object: query, graded relevant (id -> 0..3), and the controlling authority. Pass --golden to switch sets.

Every eval run also captures misses into the failure store (disable with --no-failures):

uv run python -m eval.failure_loop --summary     # digest by type and harness
uv run python -m eval.failure_loop --pending     # records awaiting review
uv run python -m eval.failure_loop --review <failure_id> "note"

Citation Evaluation (CourtNav QA pairs)

Scores page-level citation grounding against the 21 lawyer-verified QA pairs of adalat-ai/Indian-Legal-Retrieval-Generation:

# one-time: extract PDFs to page records and build the page index
uv run --extra ml python -m eval.run_citation_eval --backend semantic
uv run --extra ml python -m eval.run_citation_eval --backend fused     # semantic + lexical re-ranking
uv run python -m eval.run_citation_eval --backend keyword              # baseline
uv run --extra ml python -m eval.run_citation_eval --mode doc          # document ranking

Inputs (gitignored, rebuilt from the HF dataset — see eval/run_citation_eval.py): data/hf/adalat-retrieval/ (CSV + PDFs), data/legal/courtnav/pages.jsonl (79 pages), and the LaBSE index at models/courtnav/.

Source-aware Supreme Court snapshot

The rebuilt 2000–2025 collection contains 21,529 distinct judgments/orders and 929,151 passages. It preserves every previous case identity and adds two official PDF recoveries. It remains a fixed Supreme Court snapshot; it does not cover all Indian courts or establish current precedent status. See corpus ingestion for pinned sources, checksums, duplicate handling, date discrepancies and OCR limits.

Install ingestion and model dependencies, plus system Poppler. Tesseract and Poppler's rendering utilities support the image-only fallback.

uv sync --extra ingest --extra ml --extra serve --extra llm
uv run --no-sync python scripts/rebuild_source_aware.py --years 2000-2025
uv run --no-sync python scripts/build_index.py \
  --corpus data/legal/corpus/sc_2000_2025_source_aware.jsonl \
  --chroma-dir models/chroma-source-aware --collection legal_corpus \
  --compact --batch-size 512 --max-tokens 512

The recovery command verifies the pinned archives and publishes the merged corpus only after extraction and old-case coverage checks pass. The index command builds independent lexical retrieval and dense vectors. Both commands resume verified work; the API refuses an incomplete dense build. Run the unchanged retrieval benchmark before switching the configured corpus and index together:

uv run --no-sync python -m eval.run_eval \
  --chroma-dir models/chroma-source-aware --backend fused

Use --backend both to retain the dense-only comparison as well; each backend reports its own target misses.

Set these values in .env, preserving your provider configuration:

LEGAL_AI_CORPUS_PATH=./data/legal/corpus/sc_2000_2025_source_aware.jsonl
LEGAL_AI_CHROMA_DIR=./models/chroma-source-aware
LEGAL_AI_CHROMA_COLLECTION=legal_corpus
LEGAL_AI_SOURCE_ARCHIVE_ROOT=./data/hf/scj
LEGAL_AI_RETRIEVAL_BACKEND=fused

Restart the API to reload settings. /health reports process liveness and file presence, so verify an actual research turn and source PDF after switching. The parent-text SQLite sidecar builds with bounded memory on first document read. Retain the hash-checked source archives under the configured trusted root for /document/{parent_id}/pdf to serve the extracted edition. Known archive layouts can move to a different root without rewriting the corpus.

The original sc_2000_2025.jsonl and models/chroma/ remain compatible rollback artifacts. Restore both old paths together to revert; do not pair an old corpus with the new retrieval index. New document IDs preserve verified case aliases for unambiguous legacy document links.

LLM Stage (Groq, OpenAI, NVIDIA, AgentRouter)

The deterministic floor is the default; LLM capabilities activate with LEGAL_AI_ENABLE_LLM=true plus a Groq key (GROQ_API_KEY in .env or LEGAL_AI_GROQ_API_KEY). Every LLM output is verified before it ships and every failure degrades to the deterministic path with an explicit llm_status — the floor is always the safe output.

# 1. Add your key to .env (free at console.groq.com/keys)
#    GROQ_API_KEY=gsk_...
# 2. Flip the switch
export LEGAL_AI_ENABLE_LLM=true

# 3. Offline test suite (stub clients, no network) — always works
uv run pytest tests/test_llm.py -q

# 4. Evaluate the real provider through the summarization harness
uv run --extra llm python -m eval.run_summarize_eval --help
# Use the --llm flag with the corpus/reference options documented below.

Capabilities, all behind verification gates:

• Facts pass (tools/llm_passes.extract_facts_with_llm) — LLM reads the judgment body to extract the facts narrative the SCR headnote lacks; every sentence must share a legal anchor (Art.21, s 66A, a year) with the judgment or it is dropped. Wired into the summarize node (facts_llm field on the summary).
• Hindi translation (translate_sections_with_llm) — LLM translates the structured sections; the gate requires Devanagari output and that every legal anchor number (21, 66A, 2015) survives — by number, so "Article 21" → "अनुच्छेद 21" passes. Failure falls back to the bilingual floor.
• Drafting (tools.act.draft_petition) — LLM drafts the GROUNDS from retrieved sources; the citation gate rejects any [source_id] not in the retrieved set (hallucinated authorities cannot ship), and the human-review interrupt stays.

The model ID (LEGAL_AI_LLM_MODEL, default openai/gpt-oss-120b) must be available on the selected provider and account. Provider errors fall back to source excerpts; there is no separate model-catalog validation call.

To use NVIDIA inference, install the existing LLM extra (uv sync --extra llm) and set these values in the server environment or .env:

LEGAL_AI_ENABLE_LLM=true
LEGAL_AI_LLM_PROVIDER=nvidia
LEGAL_AI_LLM_MODEL=deepseek-ai/deepseek-v4.1-flash
LEGAL_AI_LLM_FALLBACK_MODEL=
LEGAL_AI_LLM_BASE_URL=
NVIDIA_API_KEY=your-nvidia-api-key

NVIDIA defaults to https://integrate.api.nvidia.com/v1, using the OpenAI-compatible chat endpoint. LEGAL_AI_NVIDIA_API_KEY can supply an explicit scoped key instead. Clear any previous provider's fallback model and base URL when switching. An optional fallback must be a model served by the same provider; without one, answer checks use the main model. The existing completion limits, retry limits, usage ledger and answer checks apply to NVIDIA too. It uses JSON-object mode; Groq's model-specific strict-schema and reasoning parameters are not sent to NVIDIA.

To use AgentRouter, install the same LLM extra and set these values in the server environment or .env:

LEGAL_AI_ENABLE_LLM=true
LEGAL_AI_LLM_PROVIDER=agentrouter
LEGAL_AI_LLM_MODEL=deepseek-v4-flash
LEGAL_AI_LLM_FALLBACK_MODEL=
LEGAL_AI_LLM_BASE_URL=
AR_API_KEY=your-agentrouter-api-key

The configured AgentRouter default is https://agentrouter.org/v1. LEGAL_AI_AGENTROUTER_API_KEY can supply an explicit scoped key instead. Choose a model ID available to your AgentRouter account; the example ID is provider-specific and differs from NVIDIA's model ID. Clear the previous provider's fallback and endpoint when switching, or set an explicit LEGAL_AI_LLM_BASE_URL for your gateway. A fallback must be served by the same gateway. This integration uses the OpenAI-compatible chat API and sends the provider-specific User-Agent defined in tools/llm_client.py. Completion limits, bounded retries, token accounting and answer checks also apply to AgentRouter.

Serving Layer + Chat UI (Phase 9)

A FastAPI app wraps the graph and a minimal dark-mode chat UI is served at / — streaming node progress (SSE), source cards, verbatim quotes with offsets, and the human-review approve/reject flow for drafts.

uv sync --extra serve            # adds fastapi + uvicorn (plus llm/ml as needed)
uv run --extra serve uvicorn api.main:app --reload
# open http://127.0.0.1:8000

Endpoints:

• GET /health — liveness + active mode (llm on/off, retrieval backend)
• POST /chat {message, thread_id} — synchronous single-turn invoke; returns interrupted: true + review payload when drafting pauses
• POST /stream — SSE stream of per-node progress events, final done
• POST /supervise {message, thread_id} — multi-agent supervised run (Phase 8B): the supervisor LLM-plans a delegation across three real sub-agent graphs (researcher → analyst → drafter), re-invokes insufficient results with sharpened tasks, and composes the final payload; deterministic fallback when LLM is off
• GET /usage — LLM usage ledger for the process: call counts, exact token totals per model, and ESTIMATED dollar cost (point-in-time price table; token counts are the billing-grade numbers)
• GET /history/{thread_id} — accumulated turns from the checkpointer
• GET /document/{parent_id} — the whole reassembled parent judgment a retrieved chunk came from (metadata + full text + characters). Retrieval ranks chunks, so this is what lets a client show the case and not only the slice; 404 when the corpus is absent or the id is not in it, 422 for a malformed id
• POST /review/{thread_id} {approved, note} — resume an interrupted draft (approve → document generated; reject → logged, nothing drafted)

Note on Groq's free tier: the daily token budget (200k TPD) can be exhausted by long map-reduce runs — the graph then degrades to the deterministic floor with llm_status: unavailable:... until the window resets.

Current State

The whole Text-track scope is implemented: the graph (routing, checkpointing, human-in-the-loop, self-corrective retrieval), semantic retrieval over the scaled corpus, structured summarization with output variants, gated LLM capabilities (Groq), the multi-agent supervisor, the serving layer, and the evaluation suite. Run the graph with uv run python -m agent.core. The .env.example file lists every LEGAL_AI_* knob that shared/config.py reads — nothing else is consulted.