README.md

metadata

title: RingGuard
colorFrom: red
colorTo: yellow
sdk: docker
pinned: true
license: mit
short_description: Graph-based fraud ring detection for Razorpay merchants

RingGuard

Coordinated fraud detection for Razorpay merchants — finding the rings per-order scoring can't see.

RingGuard analyzes a merchant's order and account data to detect coordinated abuse — accounts secretly linked by shared devices, addresses, or payment instruments, working together to exploit returns, promo codes, or chargebacks. Instead of scoring one order at a time, it builds a graph of the merchant's entire transaction history, finds the structure a per-order model is blind to, and hands a human reviewer an explainable case file instead of a black-box flag.

Built for Track 02 — AI Risk Manager at the Razorpay AI Buildathon 2026.

---

Table of Contents

1. The Problem
2. The Solution
3. Why This, Not Thirdwatch
4. How It Works
5. AI Judgment — Where Each Tool Is Used
6. System Architecture
7. Tech Stack
8. Repository Structure
9. Getting Started
10. Usage
11. Evaluation and Metrics
12. Known Limitations
13. Policy and Safety Boundaries
14. Audit Trail
15. Demo
16. Roadmap
17. Acknowledgments
18. License

---

The Problem

Cash-on-delivery return-to-origin rates in Indian e-commerce run as high as 30-40%, and coordinated abuse — not isolated bad actors — is a meaningful share of that loss. Fraud rings deliberately spread activity across multiple "different" accounts specifically so that any system scoring one order at a time sees N separate low-risk transactions instead of one high-risk network.

Per-order risk engines, however sophisticated, are structurally blind to this by design — they were never built to look across accounts.

The Solution

RingGuard is a defensive, AI-assisted investigation agent that:

• Ingests a merchant's order, customer, device, address, and payment data (Razorpay Orders/Payments-shaped).
• Builds a graph connecting accounts through shared devices, payment instruments, addresses, and contact details.
• Detects communities in that graph — clusters of accounts more connected to each other than chance would suggest.
• Scores each cluster with a trained LightGBM model using graph structure and behavioral features.
• Explains the finding in plain language via an LLM that reads only the structured evidence it is given — never free-reasoning, never inventing facts.
• Recommends a bounded action — flag, hold, or escalate — gated by deterministic policy rules, with a human always in the loop for anything above the auto-hold threshold.

It never bans an account or seizes funds. It flags, it explains, and it waits for a human on anything that matters.

Why This, Not Thirdwatch

Razorpay has owned Thirdwatch since 2019 — a mature fraud/RTO engine scoring individual orders on roughly 300 device and behavioral signals. Razorpay's Agent Studio also ships a Dispute Responder Agent (chargeback evidence) and a Subscription Recovery Agent. None of these look across orders and accounts for structure — they operate at the level of one order or one case at a time.

RingGuard is not a better per-order scorer. It is a different problem shape: network-level detection of coordinated behavior that is invisible when you only ever look at one order at a time. That is the entire differentiation argument, and it is stated here first rather than left for a judge to have to ask about.

How It Works

Synthetic order/account stream (Razorpay Orders/Payments-shaped)
        |
        v
Entity resolution — normalize phone/email/address, hash device + payment fields
        |
        v
Graph construction — Customer / Device / Address / PaymentInstrument nodes
        |
        v
Community detection — connected components + Louvain (deterministic, seed=42)
  Two-pass: strong subgraph first (shared device/instrument >= 2 orders),
            weak-only subgraph second (shared address, no strong link)
        |
        v
Feature extraction — cluster size, strong/weak link ratio, order velocity,
                      return rate, chargeback rate, promo usage, exposure (14 features)
        |
        v
LightGBM risk model — single learned probability, no hand-blended scoring
        |
        v
Policy gate (deterministic, v3.0)
   |-- risk >= 0.85 and exposure <= Rs.25,000 ->  bounded auto-hold (Razorpay API, test mode)
   |-- risk >= 0.85 and exposure >  Rs.25,000 ->  escalate, no auto-action
   |-- 0.5 <= risk < 0.85                    ->  escalate to human review queue
   +-- risk < 0.5                            ->  log only
        |
        v
Investigator agent (LangGraph + Groq LLM) — evidence bundle in, case narrative out
        |
        v
Razorpay test-mode action (order.create + order.edit notes + order.fetch verify)
        |
        v
Audit logger — every stage, every decision, timestamped (audit_log.jsonl)

A cluster built entirely from weak links (e.g., three accounts sharing only an address — plausible for a household) is designed to score low. Distinguishing that from a strong-link cluster (shared device fingerprint used in 2+ orders) is what keeps false positives down, and it is built into the feature set from the start.

AI Judgment — Where Each Tool Is Used

Stage	Tool	Why
Graph construction, edge rules, policy gate	Deterministic code	Money-adjacent decisions need to be boring and auditable — no model judgment here
Community detection, clustering	Classical graph algorithms (Louvain + connected components)	Structural pattern detection, not a language problem
Ring risk scoring	LightGBM (supervised, tabular)	Faster, cheaper, and more auditable than an LLM for a numeric classification task
Case narrative and explanation	LLM (Groq — openai/gpt-oss-120b)	The one place where synthesizing evidence into something a reviewer can act on quickly genuinely benefits from language generation
Final action	Deterministic policy gate, human-approved above threshold	The LLM never has authority to move money or take action on its own

System Architecture

flowchart TD
    A[Synthetic Order Stream] --> B[Entity Resolution]
    B --> C[Graph Construction - NetworkX]
    C --> D[Two-Pass Community Detection]
    D --> E[Feature Extraction - 14 features]
    E --> F[LightGBM Risk Model]
    F --> G{Deterministic Policy Gate}
    G -->|risk >= 0.85, exposure <= 25k INR| H[Auto-Hold via Razorpay API]
    G -->|risk >= 0.85, exposure > 25k INR| I[Escalate to Human Review]
    G -->|0.5 <= risk < 0.85| I
    G -->|risk < 0.5| J[Log Only]
    H --> K[Investigator Agent - LangGraph + Groq LLM]
    I --> K
    K --> L[Case Narrative + Confidence Statement]
    L --> M[Audit Log + FastAPI Dashboard]
    J --> M

Tech Stack

Layer	Choice
Language / API	Python 3.11, FastAPI
Graph	NetworkX + python-louvain
Scoring	LightGBM
Agent orchestration	LangGraph
LLM	Groq (openai/gpt-oss-120b, plain chat completion — no tool calls)
Razorpay integration	Official Razorpay Python SDK v2.0.1 (test-mode, verified)
Synthetic data	Faker
Dashboard	Server-side HTML rendered by FastAPI
Deployment	HuggingFace Spaces (Docker)

No paid dependencies anywhere in this stack.

Repository Structure

ringguard/
|-- README.md
|-- requirements.txt
|-- .env.example
|-- Dockerfile
|-- data/
|   |-- generate_synthetic_data.py     <- 500 customers, 10 rings, seed=42
|   |-- augment_demo_dataset.py        <- adds RING11 + BOUNDARY01
|   +-- synthetic/
|       |-- customers.json             <- 506 customers
|       |-- orders.json                <- 1,215 orders
|       |-- ground_truth.json          <- 11 rings
|       |-- candidate_clusters.json    <- 28 candidate clusters
|       |-- scored_clusters.json       <- 28 clusters with risk_probability
|       +-- cases_demo.json            <- 28 cases with LLM narratives + Razorpay action_taken
|-- graph/
|   |-- build_graph.py                 <- entity resolution + heterogeneous graph
|   +-- detect_communities.py          <- two-pass community detection
|-- scoring/
|   |-- features.py                    <- 14-feature extraction
|   |-- train_model.py                 <- LightGBM training (training dataset only)
|   |-- score_demo.py                  <- scores demo dataset
|   +-- model.lgb                      <- trained model (do not retrain without reason)
|-- agent/
|   |-- policy.py                      <- deterministic policy gate (v3.0)
|   |-- investigator.py                <- Groq LLM investigator node
|   |-- graph.py                       <- LangGraph StateGraph orchestration
|   +-- run_demo.py                    <- end-to-end demo runner
|-- integration/
|   +-- razorpay_actions.py            <- Razorpay test-mode: create + edit + fetch orders
|-- api/
|   +-- main.py                        <- FastAPI dashboard (/, /cases, /health)
|-- eval/
|   |-- evaluate.py                    <- holdout evaluation script
|   +-- eval_results.json              <- real evaluation numbers (do not edit manually)
+-- audit/
    |-- logger.py                      <- append-only structured audit logger
    |-- generate_audit_log.py          <- bootstraps audit_log.jsonl from cases_demo.json
    +-- audit_log.jsonl                <- full audit trail, 28 entries

Getting Started

Prerequisites

• Python 3.11+
• A free Groq API key
• Razorpay test-mode API keys (rzp_test_...) from the Razorpay Dashboard

Installation

git clone https://github.com/rohitdecodes/ringguard.git
cd ringguard
python -m venv venv
source venv/bin/activate       # Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env            # fill in GROQ_API_KEY, RAZORPAY_KEY_ID, RAZORPAY_KEY_SECRET

Environment variables

GROQ_API_KEY=your_groq_api_key
RAZORPAY_KEY_ID=rzp_test_...
RAZORPAY_KEY_SECRET=...

Keys are read from .env at runtime. .env is gitignored — never commit credentials.

Usage

The pipeline runs in order. Each step can be re-run independently.

# 1. Generate base synthetic data (seed=42, deterministic)
python data/generate_synthetic_data.py

# 2. Augment with RING11 (auto_hold case) + BOUNDARY01 (hard ambiguous case)
python data/augment_demo_dataset.py

# 3. Build the heterogeneous graph
python graph/build_graph.py

# 4. Detect candidate clusters (two-pass: strong + weak-only)
python graph/detect_communities.py

# 5. Score every cluster with the trained LightGBM model
python scoring/score_demo.py

# 6. Run the full LangGraph agent pipeline (LLM calls for escalate/auto_hold cases)
python agent/run_demo.py

# 7. Execute Razorpay test-mode actions for all actioned cases
python integration/razorpay_actions.py

# 8. Regenerate audit log from finalized cases
python audit/generate_audit_log.py

# 9. Run held-out evaluation
python eval/evaluate.py

# 10. Launch the dashboard
uvicorn api.main:app --port 7860
# Open http://localhost:7860

Deployed dashboard: https://ogrohit-ringguard.hf.space
Health check: https://ogrohit-ringguard.hf.space/health

Evaluation and Metrics

Evaluated on a held-out cluster-level split (no ring's accounts appear in both train and test). Fixed seed=42, stratified 80/20 split, 22 held-out clusters.

Metric	Value
Precision	1.0000
Recall	1.0000
F1	1.0000
False-positive cost (held orders wrongly flagged)	Rs. 0.00
Inference latency (feature extraction + model)	< 0.1 ms/cluster

Holdout breakdown: 12 positive (fraud rings), 10 negative (benign clusters).

Policy action breakdown — holdout set

Action	Count
auto_hold	0
escalate	12
log_only	10

All 12 positive clusters score >= 0.85 but have total exposure above Rs. 25,000 (realistic ring sizes accumulate large amounts), so they route to escalate rather than auto_hold. This is correct behavior.

Demo dataset — 28 clusters (post-augmentation)

Action	Count	Notes
auto_hold	1	RING11 — risk 98.67%, exposure Rs. 19,800
escalate	10	Original 10 rings — risk 99.1-99.7%, large exposure
log_only	17	Includes BOUNDARY01 (see Known Limitations)

Feature importances (top 5)

Feature	Importance
avg_order_amount	145
chargeback_rate	120
strong_link_count	90
return_rate	73
order_velocity	37

Known Limitations

Read this before drawing conclusions from the precision/recall numbers above.

The current P=R=F1=1.0 figures reflect that the synthetic training data does not yet contain genuinely ambiguous examples. The model has not been tested on a hard case — one where real human experts would disagree.

1. Training distribution boundary. The model's top feature is avg_order_amount (importance=145). Positive training examples have avg_order_amount >= Rs. 3,249; all negatives are below Rs. 3,171. This creates a near-hard threshold in a learned model. Real rings could fall on either side of it.

2. BOUNDARY01 — the designed hard case. BOUNDARY01 (CUST0504-CUST0506) was rebuilt in Phase 3 as a genuine structural challenge: 3 accounts sharing device DEV0402 with 2 orders each (count >= 2, forming a strong_anchored cluster with strong_link_count=3 — the same structural signature as a confirmed fraud ring). Yet: zero chargebacks, low return rate (1/9), modest avg_order_amount (~Rs. 1,183). Model output: risk=2.08%, log_only.

The model is not wrong. Zero chargebacks and low order values legitimately suppress the score — that is precisely what the feature weights say to do. But this also means: if a real low-value shared-device cluster were fraudulent, the current model would miss it. This is an honest limitation of the training distribution.

3. Synthetic data only. All evaluation is against Faker-generated data. Real merchants have messier entity resolution, more diverse ring structures, and ambiguous cases the current training data cannot represent. Real-world P/R/F1 should be expected to be materially lower.

4. Auto-hold threshold calibration. The original Rs. 5,000 auto-hold cap was incompatible with the training distribution (a minimal ring totals ~Rs. 19,800). Updated to Rs. 25,000 in Phase 3 (v3.0). The threshold was never validated against real production exposure data.

5. LLM model substitution. llama-3.3-70b-versatile (planned) was unavailable on the Groq account. groq/compound-mini was used first; the final pipeline uses openai/gpt-oss-120b. Both produce valid narratives with zero grounding failures across all 11 LLM calls. Model is configurable via GROQ_MODEL environment variable.

Real-world performance validation requires labeled production data. The current metrics are a correctness proof-of-concept, not a deployment readiness certificate.

Policy and Safety Boundaries

RingGuard is defense-only, per Track 02's rules:

• It can flag, hold, and escalate. It cannot ban accounts, seize funds, or take any irreversible action.
• Auto-hold only fires at risk >= 0.85 AND total cluster exposure <= Rs. 25,000. Everything above that threshold, or in the mid-confidence band (0.5-0.85), goes to a human review queue — never auto-actioned.
• The LLM has no authority to trigger an action. It explains a decision the deterministic policy gate already made.
• The LLM (openai/gpt-oss-120b) is called as a plain chat completion with no tools parameter — it has no web-search or code-execution access. It reads only the structured evidence bundle.
• All Razorpay actions are test-mode only (rzp_test_... keys enforced in code). No real money moves.

Audit Trail

Every case writes one JSON record to audit/audit_log.jsonl containing:

case_id, timestamp, cluster_members, evidence_bundle, graph_score_inputs (all 14 features),
lightgbm_probability, llm_narrative, llm_confidence_statement, policy_decision,
action_taken (Razorpay order data or null), human_reviewer, outcome,
reason_if_not_auto_actioned

The log is append-only. Every case in cases_demo.json has a matching audit entry, including log_only cases. The dashboard at /cases/{case_id}/detail shows the full audit chain for each case.

Demo

Live deployment: https://ogrohit-ringguard.hf.space

The dashboard shows all 28 detected cases, risk scores, policy decisions, and Razorpay order IDs. Click any case row to see the full evidence bundle, LLM narrative, and audit chain.

To run the pipeline locally:

# Run the full detection pipeline (LLM calls included)
python agent/run_demo.py

# Run live evaluation (produces real numbers, not pre-computed)
python eval/evaluate.py

# Start the dashboard
uvicorn api.main:app --port 7860

Roadmap

• Phase 1 — Detection Core: synthetic data generation, entity resolution, graph construction, two-pass community detection (strong + weak subgraphs). 10/10 rings recovered, 0/5 false-positive stress clusters.
• Phase 2 — Intelligence Layer: 14-feature extraction, LightGBM risk model (P=R=F1=1.0 on holdout), LangGraph agent orchestration, Groq LLM investigator node, deterministic policy gate, grounding check.
• Phase 3 — Integration and Submission: RING11 (auto_hold demo case) + BOUNDARY01 (hard ambiguous case), Razorpay Python SDK test-mode integration (create/edit/fetch, notes verified), append-only audit logger, FastAPI dashboard, HuggingFace Spaces deployment (Docker), final evaluation.

Acknowledgments

Built with LangGraph, Groq (openai/gpt-oss-120b), LightGBM, NetworkX, python-louvain, and the Razorpay Python SDK.

License

MIT — see LICENSE.